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This  article  summarizes  research  conducted  to  determine  the  use  of  Advanced 
Warfighting  Experiments  to  support  material  acquisition  decisions.  Specifically, 
the  research  evaluated  the  effectiveness  of  the  Army  Task  Force  XXI  Advanced 
Warfighting  Experiment  (TF  XXI  AWE)  objective  of  providing  information  to 
support  investment  decisions  and  refinement  of  requirements  for  emerging 
technology  initiatives.  Data  were  collected  from  appropriate  program  offices  and 
user  representatives  to  determine  the  perceived  utility  of  the  recommendations 
and  level  of  implementation.  Subjective  data  detailing  why  specific  recommenda¬ 
tions  were  or  were  not  implemented  were  used  to  determine  the  contributing  factors 
to  a  program’s  ability  to  benefit  from  participation  in  the  experiment. 
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program  managers — but  it  is  one  that  is  vastly  under-used.  This  article  provides  a 
business-case  framework  (a  methodology  to  evaluate  investment  opportunities) 
for  program  managers  within  the  Department  of  Defense  to  use  when  determining 
how  to  apply  modeling  and  simulation  in  project  management. 
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The  United  States  has  changed  its  military  strategy  and  stepped  up  the  use  of  its 
existing  military  forces  without  a  major  defense  budget  increase.  A  host  of  new 
initiatives  are  under  way  to  generate  the  cost  savings  necessary  to  continue  force 
modernization  without  a  major  budget  increase.  Reforms  so  far  have  primarily 
focused  on  the  acquisition  and  logistic  parts  of  the  problem.  Financial,  contractual, 
and  sustainment  reforms  are  needed  in  order  for  acquisition  logistics  reform  (ALR) 
to  achieve  its  full  potential. 
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success  relies  upon  identification  and  use  of  viable  performance,  cost,  schedule, 
and  risk  “trade  space  ”  The  Air  Force  has  integrated  CAIV  concepts  with  those  in 
the  Reduction  in  Total  Ownership  Program  (R-TOC),  and  has  published  a 
comprehensive  guidebook  for  better  understanding. 
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THE  USE  OF  ADVANCED 
WARFIGHTING  EXPERIMENTS 
TO  SUPPORT 
ACQUISITION  DECISIONS 

Kenneth  Strayer,  Thomas  Hoivik,  and  Susan  Page  Hotevar 

This  article  summarizes  research  conducted  to  determine  the  use  of  Advanced 
Warfighting  Experiments  to  support  material  acquisition  decisions.  Specifically, 
the  research  evaluated  the  effectiveness  of  the  Army  Task  Force  XXI  Advanced 
Warfighting  Experiment  (TF  XXI  AWE)  objective  of  providing  information  to 
support  investment  decisions  and  refinement  of  requirements  for  emerging 
technology  initiatives.  Data  were  collected  from  appropriate  program  offices 
and  user  representatives  to  determine  the  perceived  utility  of  the  recom¬ 
mendations  and  level  of  implementation.  Subjective  data  detailing  why  specific 
recommendations  were  or  were  not  implemented  were  used  to  determine  the 
contributing  factors  to  a  program’s  ability  to  benefit  from  participation  in  the 
experiment. 


From  March  1996  through  October 
1997,  the  Army  conducted  the  Task 
Force  XXI  Advanced  Warfighting 
Experiment  (TF  XXI  AWE),  culminating 
with  a  live  exercise  at  the  National  Train¬ 
ing  Center,  during  Rotation  97-06,  March 
1997.  The  purpose  of  the  AWE  was  to 
provide  sufficient  data  to  validate  digitiz¬ 
ing  the  battlefield  and  support  credible  as¬ 
sessments  on  which  to  base  future  pro¬ 
curement  decisions.  The  TF  XXI  AWE 
was  meant  to  be  a  tool  for  resolving  issues 
and  reducing  risk  early  in  the  program 
development  process  and  determining  the 


adequacy  of  requirements,  design,  and 
new  system  capabilities  before  commit¬ 
ting  major  resources.  A  total  of  93  TF  XXI 
emerging  technology  initiatives  were 
initially  included  in  the  AWE. 

The  stated  TF  XXI  AWE  objectives  in¬ 
clude  experimenting  with  advanced  tech¬ 
nologies  and  providing  information  to 
support  investment  decisions  on  the  most 
promising  initiatives  (Department  of  the 
Army,  1996).  The  AWE  was  also  meant  to 
help  the  U.S.  Army  Training  and  Doctrine 
Command  (TRADOC)  refine  requirements 
and  develop  solutions  for  Force  XXL  The 
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final  Live  Experiment  Assessment  Report, 
prepared  by  the  U.S.  Army  Operational 
Test  and  Evaluation  Command  (OPTEC), 
included  full  assessments  of  most  partici¬ 
pating  TF  XXI  initiatives.  OPTEC  pro¬ 
vided  observations  and  specific  develop¬ 
mental  recommendations  for  each  initia¬ 
tive  (Department  of  the  Army,  1997). 

AWE  as  a  Formative 
Evaluation  Process 


Advanced  Warfighting  Experiments  are 
limited  in  their  ability  to  predict  real-world 
outcomes,  since  experimental  data  gener¬ 
ally  comes  from  single  or  few  unrepeat¬ 
able  events.  Safety  restrictions,  lack  of 
realism,  and  unknown  composition  of 
future  threats  make  AWE’s  weakly  pre¬ 
dictive  at  best.  But  experimentation  is  not 
limited  to  the  rigorous  demands  of 
summative  testing;  it  includes  discovery 
learning.  Experiments  can  be  viewed  as 
“formative  exercises  to  see  what  works 
and  what  doesn’t”  (Lickteig,  1996,  p.  15). 
The  U.S.  Army  Research  Institute  recom¬ 
mended  that  the  Army  implement  a  for¬ 
mative  evaluation  method  that  focuses  on 
exploration,  explanation,  and  improve¬ 
ment.  Formative  studies  are  defined  as 
“evaluative  activities  undertaken  during 
the  design  and  pretesting  of  programs  to 
guide  the  design  process”  (Rossi,  1993, 
p.  104).  Formative  evaluations  are  con¬ 
ducted  in  developmental  stages  to  help 
form  or  improve  the  system  for  the  user 
(Lickteig,  1996). 

Researchers  at  the  RAND  Corporation 
have  demonstrated  that  the  “credible  uses” 
(CU)  framework  (Dewar,  Bankes,  Hodges, 
Lucas,  Saunders-Newton,  and  Vye,  1996), 
based  on  a  decision-to-experiment  ladder 


(DEL),  can  improve  the  experimental  re¬ 
sults  from  an  AWE  (Lucas,  Banks  and 
Vye,  1998;  Lucas,  Moore,  and  Vye,  1998). 
This  approach  requires  significant  up¬ 
front  analysis  to  link  the  experimental 
design  to  critical  programmatic  decisions. 
The  decision-to-experiment  ladder  in¬ 
cludes:  issues,  decisions  to  be  made, 
argument  to  support  the  decisions,  hypoth¬ 
eses  to  be  adjudicated,  experiments  to 
resolve  hypotheses,  and  analysis  and  mea¬ 
sures  to  implement  decisions  (Lucas, 
Banks,  and  Vye,  1998).  Examples  of  the 
ways  in  which  the  DEL  can  support 
formative  assessments  follow: 

•  The  CU  process  requires  that  specific 
decisions  be  identified  that  can  be 
resolved  by  experimentation.  Specific¬ 
ity  in  stating  focal  issues  is  an  initial 
requirement  to  experimental  design 
(Lucas,  Moore,  and  Vye,  1998). 

•  Through  understanding  the  decisions 
to  be  made,  analysts  know  exactly  the 
strength  of  the  argument  needed  to 
make  the  decision.  This  identification 
of  arguments  to  support  decisions  is 
critical  in  the  design  of  the  experiment 
and  is  much  more  productive  than 
conducting  the  experiment  and  then 
deducing  what  can  be  inferred  from  the 
evidence  (Lucas,  Banks,  and  Vye, 
1998). 

•  For  the  experiment  to  be  used  credibly, 
it  must  be  designed  to  support  deci¬ 
sions.  Because  of  the  limited  data  gath¬ 
ered  from  AWEs,  there  must  be  an 
“objective  and  traceable  link  from 
experimental  results  to  decisions  on 
important  issues”  (Lucas,  Banks,  and 
Vye,  1998,  p.  16). 
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Rossi  (1993)  provides  additional  sup¬ 
port  for  the  importance  of  the  formative 
component  of  experiments.  He  argues 
that,  no  matter  the  complexity  or  integ¬ 
rity  of  the  scientific  process  of  the  exer¬ 
cise,  the  worth  of  evaluations  must  be 
judged  by  their  utility.  To  achieve  maxi¬ 
mum  benefit  from  experiments,  the  actual 
evaluation  must  be  tailored  to  the  specific 
program,  the  stage  of  activity  in  the  pro¬ 
gram,  and  the  needs  of  the  stakeholders 
in  the  program.  Rossi  (1993)  outlines  five 
factors  that  affect  the  utilization  of  evalu¬ 
ation  findings:  relevance  to  the  problem, 
communication  between  researchers  and 
users,  information  processing  by  users, 
plausibility  of  research  results,  and  user 
involvement  or  advocacy. 

The  assessment  of  the  utility  of  data 
from  the  TF  XXI  AWE  to  Army  program 
managers  is  the  central  point  of  investi¬ 
gation  in  this  research.  In  particular, 
OPTEC  provided  specific  recommenda¬ 
tions  for  each  participating  emerging  tech¬ 
nology  initiative  in  its  Live  Experiment 
Assessment  Report.  These  recommenda¬ 
tions  detailed  changes  to  user  require¬ 
ments,  desired  technology  improvements, 
and  integration  issues.  This  research 
evaluates  the  utility  of  the  AWE  to  pro¬ 
gram  mangers  by  investigating  the  levels 
at  which  these  recommendations  were 
implemented.  Specific  factors  affecting 
utilization  are  identified  to  improve  the 
formative  benefits  of  future  AWEs. 


Research  Methodology 


The  research  methods  were  designed 
to  evaluate  the  effectiveness  of  the  TF  XXI 
AWE  objective  of  providing  information 
to  support  investment  decisions  and 


refinement  of  requirements  for  informa¬ 
tion  age  technologies.  The  methodology 
included:  creation  of  tailored  surveys  for 
each  program 
initiative  and 


dissemination 
to  each  appli¬ 
cable  program ;  : 
office  and  user,  I 


"...no  matter 
the  complexity 
or  integrity  of  the  ^ 
scientific  process 
,  ,  n  m  of  the  exercise,  the 
and  followup  wor||,  pf  evaluations 
interviews  of  mu#l  be  judged  by 
pertinent  pro- 1  their  utility." 
gram  managers, 
users,  and  test 

officials.  The  primary  research  questions 
addressed  were: 


•  Were  the  specific  recommendations  de¬ 
rived  from  the  Task  Force  XXI  AWE 
used  to  support  investment  decisions 
and  to  refine  requirements  of  partici¬ 
pating  initiatives? 

•  What  were  the  contributing  factors  to 
a  program’s  ability  to  benefit  from 
participation  in  the  AWE? 

•  What  are  the  characteristics  of  pro¬ 
grams  that  are  best  positioned  to  gain 
valued  investment  and  requirements 
information  from  participation  in 
AWE? 

A  statistical  analysis  of  collected  data 
was  conducted  to  evaluate  implementation 
of  initiative  recommendations  and  to  iden¬ 
tify  contributing  factors  to  a  program’s 
ability  to  benefit  from  participation  in  the 
AWE.  Additionally,  a  summary  descrip¬ 
tion  of  the  characteristics  of  a  program  that 
is  best  positioned  to  participate  in  future 
AWEs  was  developed. 
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The  scope  of  this  research  was  limited 
to  the  1997  TF  XXI  AWE.  The  research 
primarily  concentrated  on  material  acqui¬ 
sition  issues,  the  immediate  use  of  OPTEC 
recommendations  for  initiative  develop¬ 
ment,  and  the  contributing  factors  behind 

the  use  or  non¬ 
use  of  those 


"The  survey 
was  designed 
to  collect  both 
obiecthre  and 
subjective  data." 


recommenda¬ 
tions.  Of  the  93 
emerging  tech¬ 
nology  initia¬ 
tives  included 
in  the  experi¬ 
ment,  36  initia¬ 
tives  were  in¬ 


cluded  in  the  research.  Certain  initiatives 


were  excluded  from  the  research  for  the 
following  reasons: 


•  the  initiatives  were  not  evaluated  by 
OPTEC  during  the  AWE; 


•  OPTEC  did  not  provide  any  substan¬ 
tive  recommendations  by  which  to 
measure  implementation; 


•  the  initiative  was  a  doctrinal  or  orga¬ 
nizational  change  with  no  material 
program;  or 


•  the  program  was  subsequently  termi¬ 
nated  after  the  AWE  and  no  represen¬ 
tatives  could  be  found  to  provide  input 
towards  the  research. 


Two  representatives  from  each  of  the 
36  emerging  technology  initiatives  were 
sought  for  participation.  First,  program 
managers  were  identified  based  on  their 
direct  experience  with  the  AWE  and  their 
ability  to  provide  programmatic  insights 
into  the  process  and  results  of  the  AWE. 


Secondly,  a  user  representative  was 
identified,  in  most  cases  the  combat  de¬ 
veloper  responsible  for  the  program.  Of 
the  72  possible  representatives,  67  were 
actually  selected  to  participate  in  this 
study. 

Survey  Design 

The  survey  was  designed  to  collect  both 
objective  and  subjective  data.  The  objec¬ 
tive  survey  items  assessed  the  specific 
level  of  implementation  of  recommenda¬ 
tions  made  in  the  Live  Experiment  Assess¬ 
ment  Report.  Subjective  items  were  in¬ 
cluded  to  gather  perceptions  and  opinions 
from  specific  program  offices  and  user 
representatives.  The  subjective  items  were 
structured  from  the  RAND  Corporation's 
CU  framework  (Dewar  et  al.,  1996)  and 
the  decision-to-experiments  ladder  (DEL) 
that  directly  links  experiments  to  decisions 
(Lucas,  Moore,  and  Vye,  1998).  Specific 
survey  items  were  framed  on  the  decision- 
to-experiment  ladder  to  allow  participants 
to  demonstrate  through  their  responses  the 
extent  to  which  this  decision  ladder  was 
implemented  during  the  AWE. 

Survey  Instrument 

The  survey  instrument  utilized  a  com¬ 
bination  of  ordinal  measurements  reflect¬ 
ing  the  level  of  implementation  achieved 
for  each  OPTEC  recommendation  and 
factors  affecting  implementation,  as  well 
as  open-ended  questions  designed  to 
gain  subjective  perceptions  about  the 
AWE  process.  The  survey  contained  four 
sections. 

Section  1  asked  respondents  to  rate  the 
level  of  implementation  of  the  specific  rec¬ 
ommendations  made  for  their  program 
initiative  by  the  Army  OPTEC’s  Live  Ex¬ 
periment  Assessment  Report.  Participants 
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were  asked  to  indicate  the  term  that  best 
described  the  extent  that  the  recommen¬ 
dation  was  implemented:  fully,  mostly, 
limited,  not  at  all.  Participants  were  also 
asked  to  provide  narrative  comments 
explaining  the  factors  that  influenced  the 
degree  of  implementation. 

Section  2  contained  questions  about  the 
program’s  experiences  in  the  1997  TF 
AWE  that  were  derived  from  the  decision- 
to-experiment  ladder  (Lucas,  Moore,  Vye, 
1998).  The  questions  covered  all  of  the 
applicable  components  from  issue  devel¬ 
opment  to  analysis  and  implementation. 
Participants  were  asked  to  rate  the  follow¬ 
ing  three  questions  using  a  five-point  scale 
(with  1  being  low,  5  being  high): 

•  To  what  extent  were  you  able  to  tailor 
or  influence  your  program’s  specific 
activities  in  the  AWE  to  relate  to  the 
issues  and  decisions  you  faced  as  an 
acquisition  manager? 

•  How  valuable  were  the  data  and  rec¬ 
ommendations  gained  from  participa¬ 
tion  in  the  AWE  in  making  decisions 
as  an  acquisition  manager? 

•  To  what  extent  did  your  program 
benefit  from  participating  in  the  AWE? 

Respondents  also  were  asked  to  give  a 
brief  narrative  explanation  of  each  rating. 
In  addition  to  the  objective  ratings,  the 
following  three  open-ended  questions 
were  included  to  gather  contributing  fac¬ 
tors  and  characteristics  of  programs  best 
positioned  to  benefit  from  AWEs: 

•  What  were  the  specific  developmental 
ISSUES  being  addressed  on  your  pro¬ 
gram  at  the  time  of  its  participation  in 


the  AWE  and  what  decisions  were  to 
be  made  from  gathered  data? 

•  What  were  the  contributing  factors  to 
your  program’s  ability  to  benefit  from 
participation  in  the  AWE? 

•  Based  on  your  program’s  experience 
in  the  1997  AWE,  describe  the  charac¬ 
teristics  of  a  program  that  would  best 
be  situated  to  benefit  from  participation 
in  a  future  AWE. 


Section  3  gathered  respondent  opinions 
on  reasons  why  recommendations  made 
by  the  1997  AWE  were  or  were  not  fully 
implemented.  The  questionnaire  at¬ 
tempted  to  include  an  exhaustive  list  of 
potential  impediments  or  positive  factors 
for  implementation.  Participants  were 
asked  to  rate 
each  of  the 
items  on  a  five- 
point  scale  (1, 1 
not  significant; 

5,  highly  sig¬ 
nificant).  Ex-  f 
amples  of  rea- 
sons  for  less 
than  full  imple- 

mentation  included:  lack  of  money;  lack 
of  time;  questionable  validity  of  test  data 


"The  questionnaire 
attempted  to  include 
an  exhaustive  list 
of  potential  impedi¬ 
ments  or  positive 
factors  for 
implementation." 


and  recommendations;  and  technical  fea¬ 
sibility.  Examples  of  factors  supporting 
implementation  included:  high  priority  by 
users,  high  priority  of  program  office, 
available  funding. 

Section  4  asked  respondents  to  provide 
any  additional  comments  on  their 
program’s  participation  in  the  AWE,  in¬ 
cluding  any  information  that  might  assist 
acquisition  managers  in  gaining  maxi¬ 
mum  benefit  from  participation  in  future 
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experiments  or  that  would  assist  planners 
in  tailoring  future  experiments  to  better 
benefit  participating  programs. 

Interviews 

Finally,  interviews  were  conducted  with 
nine  program  representatives  in  order  to 
solicit  more  detailed  information  address¬ 
ing  some  of  the  questions  in  the  survey. 
Examples  of  questions  included  in  the 
interview  are: 

•  How  did  your  program  benefit  from 
participating  in  the  AWE? 

•  What  kind  of  specific  feedback  did  you 
receive  on  your  program’s  perfor¬ 
mance  in  the  AWE?  Was  it  valid? 
Could  you  take  advantage  and  utilize 
the  feedback? 

•  How  did  the  maturity  of  your  program 
affect  its  performance  at  the  AWE? 

•  Were  you  able  to  tailor  the  analysis 
plan  for  your  program  with  OPTEC? 

•  What  risks  were  involved  with  partici¬ 
pating  in  the  AWE?  How  were  those 
risks  mitigated? 

•  How  would  you  characterize  a  program 
that  would  stand  to  significantly  benefit 
from  participation? 

Respondents 

Sixty-seven  surveys  were  administered 
to  both  program  managers  and  user  rep¬ 
resentatives  of  35  different  AWE  initia¬ 
tives.  A  total  of  38  respondents  returned 
completed  surveys  for  a  response  rate  of 
56.7  percent.  Military  respondents  ranged 
from  major  (0-4)  to  colonel  (0-6)  and  had 


an  average  of  1.9  years  experience  on  the 
program  in  question.  Civilian  respondents 
ranged  from  GS-12  to  GM-15  and  had  an 
average  of  7.5  years  experience  on  their 
programs.  The  minimum  amount  of  time 
any  respondent  had  with  their  program 
was  12  months.  Also  included  were  four  civi¬ 
lian  contractors  who  directly  supported 
program  offices  or  user  agencies. 

Analysis  Strategy 


The  analysis  methodology  included  a 

review  of  the  respondent  surveys  to 

identify  the  following: 

1.  The  average  level  of  implementation 
of  all  included  initiative  recommenda¬ 
tions. 

2.  The  overall  perception  of  the  program’s 
ability  to  tailor  or  influence  the  initia¬ 
tives  specific  activities  in  the  AWE. 

3.  The  overall  perceived  value  of  the  data 
and  recommendations  gained  from 
participation  in  the  AWE  as  acquisition 
managers. 

4.  The  overall  perceived  benefit  gained 
from  participating  in  the  AWE. 

5.  The  relationship  between  degree  of 
implementation  of  recommendations 
and  factors  such  as  program  maturity, 
program  tailorability,  perceived  value 
of  the  data  and  perceived  benefit 
received  from  the  AWE. 

6.  A  hierarchy  of  reasons  why  recom¬ 
mendations  were  or  were  not  fully 
implemented. 
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To  address  these  analytic  questions, 
arithmetic  means,  medians,  and  modes 
were  computed  for  each  rated  question  to 
determine  relationships  and  effects.  A 
confidence  interval  was  computed  that 
should  include  the  true  value  of  the  pa¬ 
rameter  95  percent  of  the  time.  For  most 
analyses,  medians  were  used  for  compari¬ 
son  so  that  outliers  would  not  have  a  sig¬ 
nificant  impact  on  results.  Median  results 
for  the  various  collected  factors  were  com¬ 
pared  against  one  another  and  analyzed 
to  determine  trends  and  overall  effects  on 
the  ability  to  benefit  from  the  AWE.  The 
Kruskal-Wallis  method  was  used  to  test 
the  equivalence  (p  <  .05)  of  the  various 
factor  medians.  Finally,  the  collected  sub¬ 
jective  comments  were  analyzed  to  draw 
conclusions  to  answer  the  primary  and 
secondary  research  questions  and  develop 
a  characteristic  description  of  programs 
that  are  best  positioned  to  gain  valued  in¬ 
vestment  and  requirements  information 
from  participation  in  AWEs. 


Research  Findings 


Use  of  Recommendations  to 
Support  Investment  Decisions 

The  specific  recommendations  derived 
from  the  TF  XXI  AWE  were  used  to  sup¬ 
port  investment  decisions  and  to  refine 
requirements  of  participating  initiatives  in 
some  cases.  Overall,  programs  reported 
that  52  percent  of  the  OPTEC  recommen¬ 
dations  from  the  AWE  Live  Assessment 
Report  were  either  fully  or  mostly  imple¬ 
mented.  Thirty  percent  of  recommenda¬ 
tions  were  not  implemented  at  all.  Respon¬ 
dents  indicated  only  a  moderate  benefit 
from  participating  in  the  AWE  and  that 


the  data  and  recommendations  received 
were  only  somewhat  valuable.  The  most 
cited  reason  for  recommendations  not  be¬ 
ing  implemented  was  a  lack  of  funding. 
The  recommendation  was  most  likely  to 
be  implemented  if  it  was  a  high  priority 
of  the  user.  Participants  considered  data 
valuable  in  making  decisions  when  the 
data  met  at  least :  7 

one  of  three  f 

requirements,  g  "A  wide  range 
First,  the  data  confounding 


provided  actual 


factors  effected  an 


user  feedback 


initiative's  ability 

on  soecific  user  *°  ben®,lt  ,r#m 
onspecilicuser  fheAWE„ 

requirements. 

Second,  the  data 

contributed  to  the  development  of  tactics, 
techniques,  and  procedures.  Third,  the 
data  were  provided  at  a  time  when  it  could 
be  instrumental  in  refining  requirements 
and  design. 


Contributing  Factors  to 
a  Program's  Ability  to  Benefit 

The  survey  data  showed  that  the  level 
of  recommendation  implementation  and 
the  perceived  level  of  benefit  from  the 
AWE  were  generally  related.  It  cannot  be 
concluded,  however,  that  the  implemen¬ 
tation  of  AWE  recommendations  was 
solely  responsible  for  a  respondent’s  per¬ 
ception  of  benefit.  A  wide  range  of  con¬ 
founding  factors  effected  an  initiative’s 
ability  to  benefit  from  the  AWE. 

In  determining  the  contributing  factors 
to  a  program’s  ability  to  benefit  from  AWE 
participation,  a  program  manager  must 
first  define  the  term  benefit,  as  it  relates 
to  his  program.  Program  managers  re¬ 
ported  the  following  potential  benefits 
from  their  TF  XXI  AWE  participation: 
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•  marketing  and  exposure  of  program; 

•  early  user  feedback; 

•  refinement  of  user  requirements; 

•  development  of  tactics,  techniques,  and 
procedures  (TTP); 

•  follow-on  support  for  funding  and 
production  decisions; 

•  information  on  integration,  interfaces, 
and  interoperability;  and 

•  exposure  of  developers  to  the  user’s 
environment. 


Linkage  Between  AWE  Design  and 
Program  Issue  or  Decision.  Survey  re¬ 
sults  found  that  those  program  managers 
who  could  provide  a  detailed  explanation 
of  the  specific  programmatic  issues  being 
addressed  by  their  participation  in  the 

AWE  or  the 
decisions  to  be 
made  from 
gathered  data, 
generally  re¬ 
ported  a  much 
higher  level  of 
perceived  ben- 
efit  from  the 
experiment.  Those  programs  that  specifi¬ 
cally  developed  test  objectives  and  mea¬ 
surement  processes  for  the  AWE  that  were 
linked  to  specific  acquisition  decisions, 
were  better  positioned  to  benefit  from 
participation. 

Tailoring  of  AWE  Participation.  The 

extent  a  manager  was  able  to  tailor  or 
influence  a  program’s  specific  activities 
in  the  AWE  to  relate  to  the  program’s 


"Survey  com¬ 
ments  Indicate 
a  substantial  risk 
to  programs 
participating 
in  an  AWE." 


acquisition  issues  and  decisions  directly 
contributed  to  the  extent  that  the  program 
benefited  from  participation.  Those  initia¬ 
tives  reporting  a  high  level  of  ability  to 
tailor  or  influence  generally  reported 
much  higher  levels  of  perceived  benefit 
from  participation  in  the  AWE.  Both  pro¬ 
gram  managers  and  user  representatives 
agreed  that  to  achieve  maximum  benefit 
from  AWEs,  acquisition  managers  must 
be  able  to  participate  in  the  planning 
process. 

Program  Maturity.  Program  maturity 
has  an  impact  on  an  initiative’s  ability  to 
benefit.  While  programs  at  all  levels  of 
maturity  can  gain  from  AWE  participation, 
initiatives  in  the  mid-range  of  develop¬ 
ment  are  best  positioned  to  benefit.  These 
programs  are  sufficiently  mature  and 
rugged  enough  to  tolerate  the  harsh  envi¬ 
ronment  of  AWEs,  have  architectures  that 
are  not  yet  finalized,  and  can  make  the 
most  use  of  information  derived  from 
participation. 

Risk  Assessment.  Survey  comments 
indicate  a  substantial  risk  to  programs 
participating  in  an  AWE.  Program  risks 
include  factors  beyond  the  costs  of  par¬ 
ticipation,  to  include  a  poor  return  on  in¬ 
vestment,  potential  negative  exposure,  and 
extensive  changes  in  requirements.  The 
factors  contributing  to  risk  include:  ma¬ 
turity,  ruggedization  and  maintainability, 
funding  availability,  equipment  availabil¬ 
ity,  and  status  of  program  and  production 
decisions. 


Recommendations 


As  a  result  of  the  research  analysis,  we 
make  the  following  recommendations  for 
acquisition  managers  and  AWE  planners. 
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Acquisition  Managers 

Formulation  of  Objectives  and  Mea¬ 
surements.  To  maximize  the  potential  to 
benefit  from  AWE  participation,  acquisi¬ 
tion  managers  should  develop  objectives, 
measurement  processes,  and  data  docu¬ 
mentation  and  analysis  strategies  for  the 
AWE  that  will  inform  product  improve¬ 
ments  and  areas  for  future  investment.  The 
results  of  AWE  participation  should  be  ad¬ 
equately  documented  to  allow  the  use  of 
AWE  derived  information  and  findings 
throughout  the  developmental  life  cycle 
of  the  participating  program. 

Participation  in  the  Planning  Pro¬ 
cess.  To  achieve  maximum  benefits  from 
the  AWE,  acquisition  managers  should  ac¬ 
tively  participate  in  the  AWE  planning 
process  and  tailor  their  program's  activi¬ 
ties  in  the  AWE  to  relate  to  the  issues  and 
decisions  facing  the  program.  Data  col¬ 
lection  and  analysis  plans  should  also  be 
tailored  to  ensure  that  the  information 
derived  from  AWE  participation  is  of 
value  to  the  program. 

Risk  and  Benefit  Comparison  and 
Analysis.  Acquisition  managers  contem¬ 
plating  participation  in  future  AWEs 
should  conduct  a  detailed  analysis  of  the 
risks  associated  with  AWEs  and  consider 
their  tolerance  for  risk  as  a  factor  in  mak¬ 
ing  a  decision  to  participate.  Potential 
risks  should  be  compared  to  potential  ben¬ 
efits  from  participation  and  interpreted  to 
determine  the  best  course  of  action  for  the 
program.  Active  measures  should  be  iden¬ 
tified  to  mitigate  the  specific  risks  associ¬ 
ated  with  AWE  participation.  Methods  to 
mitigate  risk  include: 

•  early  budgeting  of  funds  for  the  AWE; 

•  marketing  to  the  user; 


•  assuring  AWE  users  are  adequately 

trained  in  the  focal  technology;  and 

•  direct  participation  in  the  AWE  planning 

process. 

AWE  Planners 

Acquisition  Manager  Involvement. 

For  the  Joint  Contingency  Force  (JCF) 
AWE,  OPTEC  representatives  should 
meet  with  program  managers  to  determine 
issues  that  are  linked  to  AWE  objectives 
(Department  of  the  Army,  1998).  Planners 
for  future  AWEs 
should  allow  ac¬ 
quisition  man- ;  "Potential 
agers  to  actively  risks  should  be 
participate  in  I  compared  to  _  _  f  jjj&y 

the  develop 
ment  of  AWE 
goals  and  ob- ! 
jectives,  scena¬ 
rios,  and  data  j 
collection  and 
analysis  plans  | 

so  that  the  information  derived  from  AWE 
participation  is  of  value  to  the  program. 

AWE  Funding.  The  Army  should  con¬ 
sider  providing  funding  for  AWE  initia¬ 
tives  so  that  acquisition  managers  can 
increase  the  benefits  derived  from  partici¬ 
pation.  Participation  in  AWEs  requires 
expensive  prototyping,  manning,  fielding, 
training,  and  transportation  costs  that  must 
be  drawn  from  existing  research  and 
development  accounts.  Program  manag¬ 
ers  cannot  increase  their  roles  in  the  AWE 
process  and  ability  to  tailor  activities  with¬ 
out  dedicated  support  from  the  Army’s 
budgetary  process. 

AWE  Program  Selection  Criteria.  The 
following  factors  should  be  considered 
when  evaluating  a  program’s  potential  to 


<■ 

potential  benefits 
front  participation 
and  interpreted  to 
determine  the  best 
course  of  action  for 
the  program." 


fb  J  “  ' v' 
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gain  valued  investment  and  requirements 
information  from  participation  in  AWEs: 

•  State  of  the  Technology  and  Goals 
of  the  AWE.  Programs  positioned  to 
receive  the  most  benefit  from  AWE 
participation  will  fit  within  the  pub¬ 
lished  goals,  objectives,  and  focus  of 


"Programs  in 
early  in  develop- 
ment  are  high  risk 
I  because  of  their 
unpredictability 
I  and  vaguely 
[  defined  roles  and 
|  requirements." 


are  not  a  high  priority  of  the  analyzing 
agency  may  not  receive  adequate  feed¬ 
back.  Additionally,  initiatives  that  are 
new  concept  technologies  without  esta¬ 
blished  current  tactics,  techniques,  and 
procedures  are  best  positioned  to  ben¬ 
efit  from  the  integration  of  multiple 
systems  in  an  experimental  environment. 

•  The  Ability  to  Tailor  Program  Par¬ 
ticipation.  The  extent  a  manager  was 
able  to  tailor  or  influence  a  program's 
specific  activities  in  the  AWE  to  reflect 
the  program’s  current  acquisition  is¬ 
sues  and  decisions  had  a  direct  impact 
on  the  extent  to  which  the  program  ben¬ 
efited  from  participating.  Those  initia¬ 
tives  reporting  a  high  level  of  ability 
to  tailor  or  influence  generally  reported 
much  higher  levels  of  perceived  ben¬ 
efit  from  participation  in  the  AWE. 


the  AWE  as 
stipulated  by 
the  planning 
officials.  Also, 
the  AWE  must 
address  the  is¬ 
sues  facing  the 
initiative.  As 
demonstrated 
in  the  TF  XXI 
AWE,  those 
programs  that 


•  Program  Participation  Objectives 
and  Strategies.  The  data  suggest  that 
those  program  managers  who  develop 
detailed  experiment  objectives  and 
expected  outcomes,  systems  for  data 
documentation  and  analysis,  and  strat¬ 
egies  for  implementation  of  AWE  data 
and  recommendations  will  receive 
more  valued  data  and  will  experience 
more  benefit  from  AWE  participation. 

•  Program  Maturity.  Program  maturity 
effects  the  ability  of  a  program  to  per¬ 
form  adequately  as  well  as  the  ability 
of  the  initiative  to  implement  recom¬ 
mendations  derived  from  the  AWE. 
Programs  in  the  mid-developmental 
phases  of  acquisition  are  best  posi¬ 
tioned  to  benefit  from  AWE  participa¬ 
tion.  Programs  in  early  in  development 
are  high  risk  because  of  their  unpre¬ 
dictability  and  vaguely  defined  roles 
and  requirements.  Programs  late  in 
development  or  in  production  are  me¬ 
dium  risk  because  they  may  be  to  far  in 
development  to  capitalize  on  recom¬ 
mendations. 

Additionally,  programs  with  no  pend¬ 
ing  production  or  funding  decisions  are 
best  positioned  to  benefit  from  AWE 
participation.  Programs  that  have 
secured  approval  and  funding  for 
production  prior  to  participation  in 
an  AWE  but  have  not  yet  begun  pro¬ 
duction,  face  the  added  risk  of  poor 
performance  and  loss  of  support.  Those 
initiatives  with  imminent  production 
decisions  are  medium  risk,  in  that  AWE 
performance  can  significantly  influence 
the  survivability  of  the  system. 
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•  System  Ruggedization  and  Main¬ 
tainability.  Systems  participating  in 
AWEs  should  be  ruggedized  and  eas¬ 
ily  maintainable.  A  system  that  is 
highly  rugged  is  able  to  withstand  the 
stresses  associated  with  operational  use 
in  harsh  environments.  A  system  that 
scores  low  in  ruggedization  has  sensi¬ 
tive  components  with  maintenance 
procedures  that  are  difficult  to  conduct 
in  a  field  environment.  Low  ruggediza¬ 
tion  may  be  associated  with  early 
prototype  systems. 

•  Program  Data  Needs  and  Require¬ 
ments.  Programs  with  extensive  need 
for  data  on  integration,  interfaces, 
interoperability,  and  user  requirements 
are  best  positioned  to  benefit.  Those 
programs  early  in  Concept  Exploration 
would  have  the  added  benefit  of  receiv¬ 
ing  early  data  and  user  feedback  to 
refine  system  requirements.  Those  pro¬ 
grams  with  sufficient  feedback  from 
other  sources,  or  not  in  a  position  to 
implement  any  recommendations  from 
the  feedback  cannot  take  full  advantage 
of  AWE  results  and  data. 

Conclusion 


The  conduct  of  AWEs  has  provided 
unique  insights  into  the  future  of  Army 


Warfare  and  the  potential  acquisition  of 
participating  programs  and  weapon  sys¬ 
tems.  However,  the  use  of  the  data  and 
recommendations  generated  to  support 
acquisition  decision  making  can  and 
should  be  increased.  The  findings  of  this 
research  strongly  reinforce  the  principles 
of  formative  evaluation  as  outlined  by 
Rossi  (1993)  and  the  Credible  Uses 
Framework  (Lucas,  Banks  and  Vye,  1998) 
that  links  experimental  design  to  explicit 
programmatic  decisions. 

Key  factors  were  identified  that  can 
enhance  the  utility  of  AWE  to  programs 
responsible  for  the  acquisition  and  devel¬ 
opment  of  technical  systems.  AWE  plan¬ 
ners  can  enhance  the  utility  of  resulting 
data  to  program  managers  by  attending  to 
specific  criteria  for  program  participation, 
involving  program  managers  in  the  experi¬ 
mental  planning,  and  providing  financial 
support  for  AWE  participation.  Program 
managers  participating  in  AWE  can  also 
take  steps  to  increase  the  utility  of  results 
through  specification  of  issues  and  deci¬ 
sions  that  can  benefit  from  data  generated 
by  the  experiment,  a  clear  assessment  of 
both  benefits  and  risks,  and  active  engage¬ 
ment  in  the  design  of  measures.  Future 
AWEs  should  focus  on  programs  that  can 
utilize  the  information  most  effectively. 
Additionally,  appropriate  funding  should 
be  provided  to  program  participants  to 
increase  their  ability  to  benefit  from  AWEs. 
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TUTORIAL 


MEASURING  THE  READINESS 
COST  OF  ONE-SIZE-SHOE- 
FITS-ALL  PUBLIC  POLICY: 

A LOOK  AT  COST-,  HYBRID-, 
AND  PRICE-BASED  PURCHASING 


Maj  Joseph  Besselman,  U.S.  Air  Force , 

Ashish  Arora,  and  Patrick  Larkey 

Analysis  of  these  three  types  of  purchasing — price-,  cost-,  and  hybrid-based — 
shows  that  a  slight  tweaking  of  current  policy  may  produce  better  results,  more 
in  line  with  commercial  buying  practices. 


The  Department  of  Defense’s  (DoD’s) 
Acquisition  Reform  team  has  fos¬ 
tered  many  promising  reforms  dur¬ 
ing  the  latter  part  of  the  1990s,  including 
the  use  of  commercial  standards,  perfor¬ 
mance-based  requirements,  integrated 
product  development,  cost  as  an  indepen¬ 
dent  variable  (CATV),  and  greater  use  of 
price-based  purchasing.  While  definitive 
evaluative  studies  do  not  yet  exist  on  the 
effects  of  these  various  reforms,  the  pre¬ 
liminary  evidence  suggests  that  greater 
use  of  price-based  purchasing  has  been  a 
mixed  blessing.  The  reform  has  apparently 
saved  the  DoD  a  great  deal  of  money  when 


used  to  buy  commodities  in  markets  where 
real  competition  exists.  However,  reform¬ 
ers  have  pushed  price-based  purchasing 
into  the  purchase  of  custom,  noncommer¬ 
cial  items  produced  under  noncompetitive 
conditions,  where  price-based  purchasing 
may  not  be  the  cost-minimizing  approach. 

The  problem  with  implementing  such 
a  far-reaching  policy,  in  general,  is  that 
without  careful  measurement,  unintended 
consequences  could  occur  and  go  largely 
undetected.  The  DoD,  and  the  federal  gov¬ 
ernment  more  generally,  have  oversight 
organizations  that  investigate  implemen¬ 
tations  of  specific  policies  on  an  ad  hoc 


The  views  expressed  in  this  paper  are  those  of  the  authors  and  not  the  official  views  of  the 
United  States  Air  Force  or  Carnegie  Mellon  University. 
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basis.  The  DoD  Inspector  General  (IG)  has 
done  at  least  four  audits  of  the  price-based 
purchasing  policy  since  1997,  uncovering 
evidence  of  widespread  overpayment  by 
the  government  for  custom,  noncommer¬ 
cial  items  (see  referenced  DoD  IG  audit  re¬ 
ports,  1998—99].  The  IG’s  contrary  evidence 
is  all  that  is  available  on  these  acquisition 
innovations  because  the  DoD  has  not  sys¬ 
tematically  measured  the  performance  of 
different  purchasing  policies. 

The  unintended  consequence  of  these 
policies,  price-based  purchasing  and  a 
lack  of  systematic  measurement,  is  that 
the  warfighter  may  be  adversely  affected 
through  higher  prices.  With  essentially 
fixed  budgets,  higher  spare  part  prices 
translate  into  fewer  spare  parts,  less  fuel, 
fewer  training  missions,  and  an  overall 

weaker  readi- 


"The  readiness 
posture  of  U.S. 
forces  is  inextrica¬ 
bly  linked  to  the 
performance  of  the 
DoD's  acquisition 
practices,  including 
the  purchase  of 
spare  parts/' 


ness  posture. 
The  readiness 
posture  ofU.S. 
forces  is  inex¬ 
tricably  linked 
to  the  perfor¬ 
mance  of  the 
DoD’s  acquisi¬ 
tion  practices, 


including  the 
purchase  of  spare  parts.  The  warfighter 
currently  takes  for  granted  what  the 
acquisition  processes  produce  by  not  im¬ 
posing  rigorous  organizational  measures 
tying  acquisition  performance  to  readiness 
posture. 

This  research  examines  the  purchase  of 
noncompetitive  spare  parts  for  exactly  the 
same  engine  bought  by  three  different  or¬ 
ganizations  using  cost-,  price-,  and  hybrid- 
based  purchasing.  The  Federal  Acquisi¬ 
tion  Regulations  (FAR  Part  15)  provide 
considerable  guidance  on  cost-  and  price- 


based  purchasing  and  allude  to  various  pos¬ 
sibilities  for  hybrid  purchasing  schemes 
depending  upon  specific  circumstances. 
According  to  FAR  Part  15,  cost-based  pur¬ 
chasing  is  the  “review  and  evaluation  of 
the  separate  cost  elements  and  profit” 
which  comprise  the  manufacture  of  an 
item,  and  “the  application  of  judgment  to 
determine  how  well  the  proposed  costs 
represent  what  the  cost  of  the  contract 
should  be”  (FAR  Part  15.404-1  [c][l]). 

A  key  strength  of  this  method  is  that 
the  collected  cost  information1  strength¬ 
ens  the  government’s  negotiating  position. 
Price-based  purchasing  is  the  “process  of 
examining  and  evaluating  a  proposed 
price  without  evaluating  its  separate  cost 
elements  and  proposed  profit”  (FAR  Part 
15.404-1  [b][l]).  It  typically  involves  de¬ 
termination  of  whether  adequate  price 
competition  exists.  It  makes  a  compari¬ 
son  of  proposed  prices  to  catalog  prices, 
historical  prices,  prices  paid  by  other  com¬ 
mercial  customers,  prices  derived  from 
parametric  estimates,  prices  from  govern¬ 
ment  cost  estimates,  and  prices  derived 
from  any  pricing  information  a  contrac¬ 
tor  may  provide  (FAR  Part  15.404-1  [b]). 
A  key  strength  of  this  approach  is  that  the 
process  requires  minimal  expenditure  of 
labor  resources  to  execute  a  purchase.  The 
hybrid-based  purchasing  method  is  an 
innovation  of  one  service.  It  capitalizes 
on  the  strengths  of  the  cost-  and  price- 
based  approaches  and  will  be  discussed 
in  greater  detail  in  the  research  findings. 

The  data  for  this  study  were  gathered 
through  site  visits  to  three  different  loca¬ 
tions:  two  buying  locations  and  the  manu¬ 
facturer.  Two  DoD  sites  support  the  reple¬ 
nishment  of  one  of  DoD’s  most  plentiful, 
multiservice  engines,  with  a  Defense  Logis¬ 
tics  Agency  (DLA)  depot  purchasing  on 
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a  cost  basis  and  a  service  depot  purchas¬ 
ing  on  a  hybrid  basis.  Their  performances 
are  compared  to  the  manufacturer’s  best  com¬ 
mercial  customer,  where  the  commercial 
customer  is  buying  on  a  price  basis.2 

Using  $142  million  in  spare  part  pur¬ 
chases  from  samples  spread  over  5  years, 
both  before  and  after  the  implementation 
of  the  DoD’s  price-based  purchasing 
policy,  the  findings  clearly  show  that  cost¬ 
and  hybrid-based  purchasing  have  re¬ 
sulted  in  substantial  savings  for  DoD. 
Across  the  samples  of  data,  the  results 
show  that  DoD  paid  $157  million  less  than 
would  the  manufacturer’s  best  commer¬ 
cial  customer  buying  on  a  price  basis 
(DoD  paid  $142  rather  than  $299  million). 
If  the  DoD  had  purchased  these  engine 
parts  at  commercial  catalog  prices,  as 
some  believe  the  DoD  should,  the  DoD 
would  have  had  to  pay  an  additional  $285 
million.  On  the  surface,  the  cost-based 
approach  appears  to  result  in  lower  cost 
acquisitions  than  the  hybrid-purchasing 
approach.  However,  the  real  picture  may 
be  more  complex.  Did  the  service’s  hy¬ 
brid-based  approach  free  up  labor  costs 
normally  devoted  to  supporting  cost-based 
purchasing?  Did  hybrid-based  purchases 
deliver  other  benefits,  such  as  better  de¬ 
livery  terms?  Did  the  cost-based  approach, 
while  getting  DoD  the  best  prices,  create 
a  more  time-  and  labor-intensive  buying 
approach?  This  article  explores  these 
issues  in  greater  detail  as  well. 

Background 


During  the  past  7  years  a  spate  of  re¬ 
forms  in  DoD  has  transformed  many  ac¬ 
quisition  processes,  using  commercializa¬ 
tion  as  the  mantra  for  many  of  the  changes. 


Ironically,  in  the  name  of  “commercial 
practices,”  price-based  purchasing  is  being 
pushed  for  buying  noncompetitive  items, 
even  though  purchasing  on  a  cost  basis  is 
a  common  commercial  practice  in  non¬ 
competitive  markets  (Perrow,  1970;  Pfeffer, 
1978;  Hardy  and  Magrath,  1987;  Burt, 
1989;  Myer,  1989;  Cross,  1995;  and  Tay¬ 
lor  and  Wiggins,  1997).  This  raises  ques¬ 
tions  about  the  judiciousness  of  the  blan¬ 
ket  application  of  price-based  purchasing 
policy  in  defense  purchasing. 

This  research  extends  earlier  work  by 
Besselman,  Arora,  and  Larkey  (1999, 2000) 
and  Besselman  (1998)  that  predicted 
pnce-based  pur- . 

chasing  of  non-;  «0n  the  surface, 
competitive  #he  cost-based 
items  would  re-  approach  appears 
suit  in  greater  i  to  result  in  lower 
financial  costs  cost  acquisitions 
to  the  U.S.  than  the  hybrid* 
Armed  Forces. purchasing 
That  research  «ppreach// 
provided  con¬ 
siderable  quantitative  evidence  that  cost- 
based  purchasing  should  be  the  method 
of  choice  when  buying  large  quantities  of 
high-value  items  in  noncompetitive  mar¬ 
kets.  To  extend  that  work,  a  survey  of 
engines  was  accomplished  to  identify 
where  cost-based  and  other  methods  of 
purchasing  are  used  to  support  a  single 
system.  It  identified  a  multiservice  engine 
that  is  sold  and  supported  by  one  com¬ 
pany.  The  DoD  owns  a  set  of  engineering 
drawings,  so  it  sometimes  uses  second¬ 
ary  sources  of  supply  for  a  small  number 
of  the  less  complex  parts  within  the  en¬ 
gine.  Although  the  manufacturer  believes 
the  engine  is  a  commercial  item,  with 
more  than  50  percent  of  its  part  sales  to 
non-DoD  customers,  the  engine’s  market 
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consists  primarily  of  militaries  of  the 
United  States  and  its  allies,  both  past  and 
present.  In  a  classic  economic  sense,  this 
engine’s  market,  with  one  supplier,  is  not 
competitive. 

The  purchase  of  the  spare  parts  to  sup¬ 
port  this  engine  by  the  DoD  is  accom¬ 
plished  at  two  locations:  a  DLA  depot  and 
a  service  depot,  although  it  is  a  multiser¬ 
vice  engine.  Until  1996,  one  service  pur¬ 
chased  all  parts  for  this  engine  on  a  cost 
basis,  both  reparable  and  consumable 
parts.  Since  that  time,  the  purchase  of 
consumables  was  transitioned  to  a  DLA 
depot.  The  DLA  depot  purchases  the 
consumables  on  a  cost-basis  while  the 


service  depot 
transitioned 
^The  use  p  the  purchase  of 

different  methods  U1  . 

by  the  two  organ!-  gables  t°  a 

zations  makes  il  hybrid-based 

possible  to  compare  approach.  This 

the  performance  of  split  in  the  sup- 

these  methods."  port  of  this  sin- 

gular  engine 
made  it  pos¬ 
sible  to  compare  the  purchasing  method¬ 
ologies  fairly.  The  use  of  different  meth¬ 
ods  by  the  two  organizations  makes  it  pos¬ 
sible  to  compare  the  performance  of  these 
methods. 


A  brief  discussion  is  required  to  distin¬ 
guish  reparable  and  consumable  spare 
parts.  Reparable  spare  parts  for  an  engine 
are  typically  more  complex  and  expensive 
on  a  per-unit  basis.  These  are  parts  that 
can  be  repaired  or  machined  and  put  back 
into  service.  The  government  typically  re¬ 
wards  manufacturers  with  greater  profit 
for  the  successful  manufacture  of  more  com¬ 


plex  items.  Complex,  reparable  engine 
parts  are  typically  purchased  in  smaller 
quantities,  thus  sometimes  negating  the 


scale  efficiencies  associated  with  the 
manufacture  of  tens  of  thousands  of 
consumables  such  as  turbine  blades. 
Within  the  service  depot  sample,  reparable 
parts  cost  on  average  more  than  $5,000 
per  item.  Consumable  parts,  on  the  other 
hand,  are  bought  in  large  quantities  to  ex¬ 
ploit  scale  efficiencies,  are  usually  sim¬ 
pler  to  manufacture,  and  are  typically  thrown 
away  or  sold  for  scrap  once  they  have  been 
used.  Within  the  depot  samples,  consum¬ 
ables  cost  on  average  approximately  $60 
per  item. 

Research  Approach _ 

We  visited  the  two  sites  and  gathered 
purchases  from  on-line  databases  and  con¬ 
tract  files.  Both  organizations  were  highly 
professional  and  cooperative,  providing 
staff  experts  to  help  gather  the  informa¬ 
tion  and  answer  contextual  questions 
about  the  purchases.  Both  sets  of  the  most 
recent  data  contained  discrepancies  that 
caused  the  removal  of  some  purchases 
from  the  analysis. 

The  service  depot  purchases  parts  us¬ 
ing  a  multiyear  blanket  purchase  agree¬ 
ment  (BPA)  with  the  manufacturer.3  The 
BPA  has  two  tiers  of  discounts  on  the  cata¬ 
log  prices:  41  percent  for  the  first  $20 
million  in  purchases  within  a  fiscal  year 
and  56  percent  for  all  volume  above  $20 
million.  According  to  the  service  depot’s 
buyer,  this  was  intended  to  collect  $3  mil¬ 
lion  from  the  DoD  to  cover  field  engineer¬ 
ing  support  services  provided  by  the 
manufacturer.4  Therefore,  to  ensure  an 
apples-to-apples  comparison,  $3  million 
to  remove  the  financial  effect  of  the  engi¬ 
neering  support  from  the  comparison  will 
adjust  the  service’s  aggregate  result  down. 
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Although  the  BPA  established  set  dis¬ 
counts,  some  purchases  had  discounts 
*  greater  and  less  than  the  prescribed 
amounts.5 

The  DLA  depot  had  no  such  purchas¬ 
ing  agreement  with  the  manufacturer.  It 
purchased  all  of  its  spares  on  a  cost  basis. 
Purchases  by  DLA  were  gathered  in  two 
samples  from  1997-98  and  1998-99. 
Since  DLA  reports  its  purchases  using 
both  in-process  and  pending  quantities,  its 
data  were  carefully  reviewed  and  only 
purchases  that  had  been  completed  were 
included  in  the  sample. 

The  analysis  of  the  purchase  data  fol¬ 
lows  a  very  simple  approach.  Purchases 
were  aggregated  and  compared  to  catalog 
prices  and  the  price  paid  by  the  manu¬ 
facturer’s  best  commercial  customer, 
where  the  best  commercial  customer  was 
buying  on  a  price  basis.  The  percent  dif¬ 
ferences  between  the  aggregate  cost  for 
each  depot  were  then  compared.  Another 
sample  of  purchases  from  the  service  de¬ 
pot  in  1995,  prior  to  the  introduction  of 
the  price-based  purchasing  policy,  is  also 
incorporated  into  the  analysis.  At  that 
time,  the  service  depot  was  purchasing  all 
spares  for  that  engine,  both  reparables 
and  consumables,  using  the  cost-based 
purchasing  method. 


Research  Results 


For  this  engine,  the  results  clearly  show 
that  buying  noncompetitive  items  on  a  cost 
and  a  hybrid  basis  is  vastly  superior  to 
price-based  purchasing.  The  DLA  data  re¬ 
vealed  that  in  1997-98  and  1998-99,  the 
DoD  paid  70  and  68  percent,  respectively, 
below  catalog  or  commercial  cost.  The 
price  difference  is  computed  by  subtract¬ 
ing  the  commercial  cost  from  the  DoD’s 
cost  and  then  dividing  the  remainder  by 
the  commercial  cost.  The  service  depot, 
however,  buying  on  a  hybrid  basis,  re¬ 
ceived  an  average  discount  of  only  63 
percent  off  the  catalog  price.  A  summary 
of  these  results  is  provided  in  Table  1. 

The  important  finding,  one  the  general 
public  does  not  expect,  is  that  the  DoD, 
regardless  of  the  purchasing  approach,  sig¬ 
nificantly  outperformed  this  manufac¬ 
turer’s  best  commercial  customer,  where 
“commercial”  in  this  sense  means  a  third- 
party  supplier  that  services  primarily  for¬ 
eign  governments  and  the  few  legitimate 
commercial  users  of  the  engine.6  Look¬ 
ing  across  the  samples  spanning  the  past 
5  years,  the  DoD  paid  $141.9  million  for 
its  spares  while  the  manufacturer’s  best 
commercial  customer,  buying  on  a  price 
basis,  would  have  paid  more  than  $299 


Table  1.  Summary  Results  ot  Engine  Purchases" 


Number  Buys 

Gov. 
Cost  ($) 

Comm. 
Cost  ($) 

Gov.-Comm. 

%  Difference 

Best  Comm. 
Customer  ($) 

Gov.-Best 
%  Difference 

1 995  Service  data 

71  (cost) 

59.3 

176.0 

-66 

123.2 

-52 

1997-98  DLA  data 

88  (cost) 

32.7 

108.1 

-70 

75.7 

-57 

1999  Service  data 

84  (hybrid) 

28.0 

75.0 

-63 

52.5 

—47 

1998-99  DLA  data 

61  (cost) 

21.9 

68.0 

-68 

47.6 

-54 

Total 

304 

141.9 

427.1 

-67 

299.0 

-53 

8  Costs  are  in  millions  of  dollars. 
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million,  a  savings  to  the  DoD  of  more  than 
$157  million. 

What  if  the  BPA  did  not  exist  and  the 
service  depot  had  to  buy  the  parts  on  a 
purely  cost  basis  (using  the  cost-based 
purchasing  method)?  Using  DLA’s  most 
recent  price  difference,  68  percent  below 
catalog  prices,  the  service  depot  could 
have  paid  only  $24.1  million  rather  than 
the  $28  million  paid,  a  savings  of  approxi¬ 
mately  $4  million.  Conversely,  had  the 
DLA  depot  used  the  BPA  and  received 

only  a  63  per¬ 
cent  discount, 
"In  theory,  cost-  the  DLA  depot 

based  purchasing  would  have 

offers  a  significant  paid  the  manu- 

advantage."  facturer  $7.6 

million  and 
$3.5  million 
more  in  1997-98  and  1998-99,  respec¬ 
tively.  The  1998-99  result  will  soon  grow 
because  DLA  had  approximately  $20  mil¬ 
lion  more  in  purchases  in  the  pipeline 
when  these  data  were  being  collected. 

All  else  being  equal,  the  purchase  of 
consumables  should  have  been  slightly 
more  effective  than  reparable  purchasing 
since  they  are  simpler  components  bought 
in  significantly  larger  volumes.  Compar¬ 
ing  the  1999  DLA  and  service  data,  con¬ 
sumable  purchasing  was  5  percent  more 
efficient  than  reparable.  Since  the  purchas¬ 
ing  methods  were  not  the  same,  this  dif¬ 
ference  cannot  be  attributed  solely  to  the 
type  of  item.  In  the  next  section  we  explore 
these  contextual  differences. 

Discussion  of  Risults 


The  findings  show  clearly  that  buying 
noncommercial  items  on  a  cost  or  hybrid 


basis  results  in  lower  costs  than  price- 
based  purchasing  buying  at  either  com¬ 
mercial  catalog  prices  or  at  the  discount 
of  the  manufacturer’s  best  “commercial” 
customer.  In  readiness  terms,  for  this  one 
engine,  the  DoD  was  able  to  keep,  depend¬ 
ing  upon  one’s  basis  of  comparison,  $157 
to  $285  million.  A  transition  to  price- 
based  purchasing  at  catalog  prices,  as 
some  officials  advocate  for  all  DoD 
engines,  would  have  adverse  fiscal 
consequences  on  readiness. 

A  lingering  issue  is  whether  the  DoD 
performed  effectively  using  the  hybrid 
method  in  relation  to  purely  cost-based 
purchasing.  The  next  section  further 
explores  their  differences  and  results. 

Purchasing  Methods 

In  theory,  cost-based  purchasing  offers 
a  significant  advantage.  It  allows  a  buyer 
with  significant  buying  power  in  a  mar¬ 
ket  to  be  near  the  informational  level  of 
the  seller.  The  buyer  with  market  power 
demands  and  receives  cost  information. 
Armed  with  good  cost  information,  the 
buyer  can  negotiate  a  price  that  holds 
down  the  seller’s  profit. 

Discussions  with  government  and  in¬ 
dustry  personnel  reveal  that  cost-based 
purchasing  also  has  its  disadvantages.  It 
is  typically  time-  and  labor-intensive  for 
the  buyer.  Indeed,  interviews  with  buyers 
as  part  of  this  research  reveal  that  time 
delays  have  increased  after  the  Federal 
Acquisition  Reform  Act  of  1996  (FARA), 
because  many  industrial  suppliers  are  try¬ 
ing  to  have  markedly  noncommercial 
items  declared  commercial  via  a  “com¬ 
mercial  catalog.”  If  an  effort  is  truly  com¬ 
mercial  or  a  contractor  is  successful  in 
persuading  a  buyer  that  it  is  commercial, 
then  the  government  is  prevented  from 
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collecting  cost  data,  no  matter  how  expen¬ 
sive  the  item  may  be.  These  efforts  by  in¬ 
dustry,  some  dubious,7  delay  negotiations 
while  the  government  determines  whether 
the  item  is  truly  commercial.  Delaying  nego¬ 
tiations  can  have  adverse  consequences  on 
readiness,  as  operational  units  may  have 
to  wait  on  an  item.  Another  labor-intensive 
feature  is  the  typical  on-site  review  of  the  con- 
tractor’s  data  and  manufacturing  processes. 

From  an  industry  perspective,  a  disad¬ 
vantageous  feature  is  the  requirement  that 
the  contractor  certify  “to  the  best  of  its 
knowledge  and  belief,  the  cost  or  pricing 
data  were  accurate,  complete,  and  current” 
[FAR  Part  15.403-4  (b)(2)].  It  can  be  fur¬ 
ther  time-  and  labor-intensive  (expensive) 
for  industry  if  the  contractor  is  also  re¬ 
quired  to  convert  its  data  to  government- 
specified  formats.  These  factors  imply  that 
the  government  carefully  consider  pur¬ 
chasing  circumstances  before  imposing  a 
purchasing  method. 

Price-based  purchasing  also  offers  sev¬ 
eral  advantages: 

•  It  can  be  accomplished  very  quickly 
(publish  a  requirement  and  solicit 
prices  from  vendors). 

•  It  uses  minimal  labor  resources. 

•  Competition,  rather  than  negotiation, 
is  used  to  control  prices. 

It  may  require  market  research  and  the 
collection  of  commercially  available  cost 
information,  but  its  transaction  costs  are 
typically  trivial  compared  to  what  it  costs 
to  collect  and  analyze  cost  information. 
In  both  the  public  and  private  sector,  price- 
based  purchasing  is  the  typical  purchase 
method  when  buying  items  competitively 


available  and  widely  used  in  the  commercial 
sector.8 


The  major  disadvantage  of  price-based 
purchasing  is  that  the  buyer  is  at  a  signifi¬ 
cant  informational  disadvantage  without 
real  competition.  In  many  markets,  mean¬ 
ingful  cost  information,  whether  certified 
or  not,  is  simply  nonexistent.  Classical 
economic  theory  requires  lots  of  suppli¬ 
ers  for  true  competition.  As  a  practical 
matter,  there  should  be  at  least  a  half  dozen 
suppliers  to  select  from  (Bresnahan  and 
Reiss,  1990, 1991).  Once  bids  have  been 
received  or  prices  surveyed,  contract 
award  is  accomplished  with  the  most 
advantageous  firm. 

The  process  quickly  breaks  down, 
however,  if  there  are  only  a  couple  of  sup¬ 
pliers.  Although  economic  theory  suggests 


that  a  single 
customer  facing 
more  than  one 
supplier  can 
drive  the  price 
down  to  mar-  ( 
ginal  cost,  eco¬ 
nomic  theory  E 
also  points  to 
the  possibility  | 
of  tacit  collu- 
sion  when  the 


"The  major 
disadvantage 
of  price-based 
purchasing  is  that 
the  buyer  is  at  a 
significant  informa¬ 
tional  disadvantage 
without  real 
competition." 


number  of  suppliers  is  small.  Further,  even 
if  the  buyer  can  drive  the  price,  if  there  is 
uncertainty  about  the  costs  of  the  suppli¬ 
ers,  economic  theory  suggests  that  the 
buyer  can  do  better  if  he  has  cost  infor¬ 
mation  rather  than  simply  relying  upon 
prices  (See  Baron  and  Meyerson,  1982; 
Grossman  and  Hart,  1983).  One  or  occa¬ 
sionally  two  suppliers  and  one  customer 
is  the  typical  circumstance  for  the  DoD 
purchasing  custom  parts  and  equipment 
for  weapon  systems.9 
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The  service  depot  used  hybrid-based 
purchasing,  an  approach  developed  to  ex¬ 
ploit  the  advantages  of  cost-  and  price- 
based  purchasing.10  They  selected  a  group 
of  parts  (reparables)  from  a  manu¬ 
facturer’s  commercial  price  list  with  the 
intent  of  putting  in  place  a  BPA  for  those 
parts.  In  this  case,  the  BPA  is  nothing  more 
than  an  agreement  by  one  government 
agency  with  a  manufacturer  to  purchase 
parts  at  either  set  prices  below  or  at  a  uni¬ 
form  discount  off  of  a  manufacturer’s 


commercial  price  list.  The  depot  then  in- 
.  vested  a  small 


"One  could 
label  the  hybrid 
approach  the  'trust 
but  verify'  theory 
of  negotiation/' 


amount  in  la¬ 
bor  to  analyze 
certified  cost 
and  pricing 
data  they  had 
previously  col¬ 


lected  for  a 


sample  of  these  same  parts.  That  analysis 
determined  an  appropriate  catalog  dis¬ 
count  equivalent  to  historical  prices  based 
on  cost  and  pricing  data.  That  analysis 
provided  an  anchor  for  negotiations.  They 
now  had  sufficient  knowledge  to  negoti¬ 
ate  a  fair  discount  on  the  manufacturer’s 


catalog  prices,  one  that  pays  a  fair  profit. 
In  the  future,  they  intend  to  collect  cost 
and  pricing  data  for  a  statistically  valid 
sample  of  the  parts  to  either  revalidate  the 
discount  or  negotiate  a  new  discount. 

The  buyers  and  engine  managers  be¬ 
lieve  the  hybrid-buying  approach  engen¬ 
ders  trust  between  the  government  and 
manufacturer  by  increasing  “contracting 
velocity,”  the  speed  at  which  a  buyer  con¬ 
tracts  for  an  item,  while  also  giving  the 
government  the  opportunity  to  verify.  The 
trust  arises  through  government  and  con¬ 
tractor  teamwork,  using  minimal  labor, 
with  very  little  contractor  disruption.  Cost 


data  is  collected  on  a  small,  representa¬ 
tive  sample  of  parts  rather  than  every 
single  item  the  government  may  purchase 
out  of  a  catalog.  This  saves  the  govern¬ 
ment  and  contractor  time  and  labor.  Veri¬ 
fication  occurs  by  basing  the  discount  on 
cost  data  for  a  statistically  representative 
sample  of  items  in  the  contract  or  cata¬ 
log.  One  could  label  the  hybrid  approach 
the  “trust  but  verify”  theory  of  negotia¬ 
tion.  Once  in  effect,  the  BPA  then  delivers 
the  strengths  of  price-based  purchasing: 
quick,  easy  purchases  with  commercial 
delivery  terms. 

Labor 

Using  the  most  recent  results,  the  ser¬ 
vice  depot,  using  the  hybrid  method,  was 
approximately  5  percent  less  efficient  pur¬ 
chasing  than  the  DLA  depot.  Are  there 
intangibles  in  hybrid-based  purchasing 
that  make  it  the  better  option,  despite  the 
greater  transaction  cost  involved?  A  major 
consideration  should  be  the  possible 
reduction  in  labor  charges  from  not  hav¬ 
ing  to  collect  and  analyze  cost  and  pric¬ 
ing  data,  both  for  the  manufacturer  and 
DoD,  and  negotiate  each  individual  con¬ 
tract.  In  this  case,  there  could  have  been 
84  separate  contract  actions  had  the  ser¬ 
vice  depot  not  established  the  BPA.  In 
actuality,  there  have  been  no  savings  from 
not  having  to  collect  cost  and  pricing  data 
for  these  purchases.  First,  the  Defense  Con¬ 
tract  Audit  Agency  (DCAA)  unit  did  not 
downsize  as  a  result  of  the  BPA.  Second, 
the  manufacturer  already  possesses  a  so¬ 
phisticated  activity-based  cost  accounting 
system  that  tracks  costs  rigorously, 
whether  or  not  cost  and  pricing  data  are 
requested  by  DoD. 

Not  having  to  negotiate  84  separate 
contract  actions  certainly  reduced  labor 
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costs.  But  these  cost  savings  are  substan¬ 
tially  less  than  the  $4  million  “loss”  the 
service  depot  experienced  by  not  using 
cost-based  purchasing.  A  contracting  of¬ 
ficer  and  four  buyers  could  have  accom¬ 
plished  the  84  contractual  actions.  Assum¬ 
ing  a  generous  labor  cost  of  $200,000  per 
staff  year,  the  total  labor  bill  would  only 
reach  $1  million,  far  below  the  estimated 
$4  million  “loss.” 

More  Favorable  Terms 

Another  consideration  is  whether  DoD 
earned  more  favorable  terms  through  the 
use  of  the  BPA.  The  BPA’s  delivery  terms 
and  lead-time  requirements  are  right  out 
of  the  commercial  catalog,  so  there  is  no 
additional  benefit.  Furthermore,  if  deliv¬ 
ery  requirements  are  shorter  than  the 
product’s  lead  time,  the  service  depot  must 
pay  the  catalog  price.  Even  when  the  DLA 
depot  violated  the  manufacturer’s  lead- 
time  requirement  buying  on  a  cost  basis, 
they  paid  a  price  near  or  above  the  cata¬ 
log  price  on  only  one  occasion.  In  effect, 
the  DLA  depot  buying  on  a  cost  basis 
enjoyed  better  delivery  terms  than  the 
service  depot. 

Emergency  Buys 

One  area  of  concern  with  the  cost-based 
buying  done  by  the  DLA  depot  is  the 
frequency  of  emergency  buys.  These  are 
buys  made  in  small  quantities  and  typi¬ 
cally  at  considerably  higher  prices  to  meet 
a  near-term  demand.  This  can  occur  be¬ 
cause  the  depot  was  notified  about  the  re¬ 
quirement  either  too  late  by  the  item  man¬ 
ager  or  its  buying  process  took  too  long 
because  of  some  inefficiency.  Although  it 
looks  bad  for  the  DoD  to  pay  $70  for  a 
turbine  blade  that  normally  costs  $19 
because  of  poor  planning  or  execution  of 


the  buy,  the  overall  effect  in  this  case  is 
not  severe  since  the  quantities  are  so  small. 
If  the  emergency  buys  were  removed  from 
the  1998-99  DLA  depot  sample,  DLA 
purchasing  would  improve  by  only  one- 
half  of  one  percent.  However,  if  that  small 
percentage  difference  were  applied  to  the 
dollar  volume  across  several  engines,  this 
inefficiency  could  pose  an  adverse  impact 
on  readiness  dollars.  Therefore,  the  DLA 
and  service  buyers  should  endeavor  to 
eliminate  emergency  buys. 

The  missing  part  of  this  analysis  is 
whether  there  was  an  adverse  readiness 
effect  because  a  part  was  not  in  the  field 
when  it  was  needed.  It  takes  time  to  buy 
on  a  cost  basis,  f 
Were  missions 

scratched  be-  "if  takes  time 

cause  the  buy-  to  buy  on  a 
ing  processes  |  cost  basis/' 
were  taking  too 
long?  In  talking 

with  the  engine’s  manager,  this  indeed 
happens,  although  not  nearly  as  often  as 
it  did  back  in  the  1997-98  time  frame.  This 
is  another  hazard  of  not  having  pervasive 
organizational  measures  for  the  leadership 
to  assess  the  costs  and  effectiveness  of  the 
buying.  The  warfighter  needs  to  rigorously 
oversee  this  process  to  ensure  purchases 
occur  on  time  and  mission  execution  is 
not  adversely  affected.  This  is  the  thread 
inextricably  linking  acquisition  and 
readiness. 

Lean  Manufacturing 

A  near-term  extension  to  this  research 
will  compare  changes  in  real  prices  over 
time.  At  this  manufacturer’s  facility,  a  sig¬ 
nificant  manufacturing  improvement  pro¬ 
gram  was  undertaken  throughout  the  early 
1990s.  The  DoD  underwrote  a  portion  of 
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this  modernization  through  the  payment 
of  higher  prices  with  the  promise  that 
prices  would  begin  to  fall  in  the  late  1990s. 
Although  inflation  has  not  been  a  factor 
in  the  markets  supplying  this  manufac¬ 
turer;  on  an  anecdotal  basis,  prices  have 
continued  to  rise.  This  is  a  concern  con¬ 
sidering  the  degree  of  innovation  that  has 
occurred  on  the  manufacturing  floor. 

This  manufacturer  has  aggressively 
pursued  innovations  commonly  referred 
to  as  lean  manufacturing.  For  example,  a 


"•••price-based 
purchasing  should 
be  the  method  off 
choice  ffor  most  off 
what  the  DoD  buys: 
commercial  items 
bought  in  competi¬ 
tive  markets." 


major  manu¬ 
facturing  cost 
driver  is  the 
length  of  the 
manufacturing 
line.  This  en¬ 
gine’s  line  has 
been  reduced 
from  more  than 
two  miles  down 


to  less  than  a 
half  mile.  In  most  normal  commercial 
markets,  particularly  when  items  are 
manufactured  over  many  years,  there  is  a 
learning  curve  that  continually  drives 
down  costs  and  consequently  prices.  At 
this  site,  the  DoD  has  not  reaped  any  of 
these  savings.  This  is  something  the  DLA 
and  service  depot  should  examine  on  an 
amicable  fact-finding  basis.  The  intent 
should  be  to  understand  why  the  manu¬ 
facturer  has  not  returned  any  savings, 
rather  than  to  put  profit  pressure  on  the 
manufacturer,  adversely  affecting  its 
financial  health. 


CONCUISIONS _ 

This  research  is  neither  an  indictment  of 
price-based  purchasing  nor  a  confirmation 


that  the  DoD  should  revert  back  to  cost- 
based  purchasing  for  all  or  most  of  its  buy¬ 
ing.  On  the  contrary,  price-based  purchas¬ 
ing  should  be  the  method  of  choice  for 
most  of  what  the  DoD  buys:  commercial 
items  bought  in  competitive  markets.  But 
like  large  commercial  firms  with  signifi¬ 
cant  market  buying  power,  DoD  should 
exploit  cost-based  purchasing  or  the  hy¬ 
brid  method  when  acquiring  noncommer¬ 
cial  items  of  significant  dollar  value  in 
noncompetitive  markets.  This  modest 
tweaking  of  the  current  policy  would  put 
the  DoD  more  in  line  with  commercial 
buying  practices. 

The  service  depot  and  other  organiza¬ 
tions  should  continue  to  refine  the  hybrid 
method.  Although  the  hybrid  method  was 
slightly  more  inefficient  (5  percent)  from 
a  cost  perspective,  the  service  depot  be¬ 
lieves  that  was  more  than  overcome  by 
awarding  contracts  quickly  and  easily,  for 
both  the  government  and  manufacturer.  In 
reality,  considering  the  service  depot  was 
buying  more  complex  components  in 
smaller  quantities,  we  expected  their  per¬ 
formance  to  suffer  slightly,  perhaps  by  a 
few  percentage  points.  Nevertheless,  the 
service  depot  should  collect  cost  and  pric¬ 
ing  data  for  a  statistically  representative 
sample  of  the  parts  they  seek  to  put  on 
contract.  The  depot  should  follow  this  up 
by  spot  checks  to  minimize  the  presence 
of  expensive  outliers. 

The  DLA  depot  should  carefully  con¬ 
sider  these  findings  and  the  benefits  they 
perceive  from  purely  cost-based  buying. 
The  service  depot  has  demonstrated  that 
it  can  get  within  5  percent  of  DLA’s  per¬ 
formance  using  the  much  simpler  and 
more  timely  hybrid  approach.  The  service 
depot  could  easily  extend  the  BPA  to 
account  for  all  parts  supporting  this  engine 
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and  free  up  valuable  DLA  labor  resources. 
A  single  BPA  for  all  parts  for  both  DoD 
organizations  would  be  a  win-win  formula 
for  government  and  the  manufacturer. 

Other  part  managers,  particularly  man¬ 
agers  of  noncommercial  items  or  systems, 
should  take  a  hard  look  at  the  hybrid 
approach.  If  one  collects  cost  data  on  sta¬ 
tistically  valid  samples,  one  should  be  able 
to  arrive  at  discounts  for  subsets  or  entire 
catalogs  that  rival  discounts  received  by 
using  cost-based  purchasing  uniformly. 
This  approach  should  free  up  valuable 
depot,  DLA,  and  Defense  Contract  Audit 
Agency  labor  resources  and  significantly 
increase  the  velocity  of  our  purchasing. 

This  research  highlights  the  most  per¬ 
vasive  commercial  practice  that  continues 
to  elude  the  DoD:  rigorous  and  wide¬ 
spread  organizational  measurement.  In  the 


DoD,  organizational  measurement  within 
and  across  much  of  what  is  called  defense 
purchasing  is  not  presently  required.  An 
organizational  measurement  initiative 
comparing  the  military  services,  buying 
sites,  internal  organizations,  and  weapon 
system  offices  would  enable  the  leader¬ 
ship  to  more  effectively  guide  defense  pro¬ 
curement  toward  genuine  process  im¬ 
provement.  This  research’s  simple  orga¬ 
nizational  measurement  exercise  showed 
that  a  $157  to  $285  million  adverse  im¬ 
pact  on  readiness  could  have  occurred  if 
the  buyers  in  the  field  had  simplistically 
implemented  a  price-based  purchasing 
policy.  Readiness,  in  general,  however,  will 
continue  to  suffer  from  inefficient  pur¬ 
chasing  until  the  leadership  embraces  co¬ 
herent  and  pervasive  measurement  across 
its  logistics  and  acquisition  enterprise. 
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Endnotes 


1.  Cost  information  is  little  more  than  a 
decomposition  of  the  unit  price,  re¬ 
vealing  the  various  constituents  such 
as  materials,  labor,  anticipated  profit, 
and  possibly  the  cost  of  money.  It  can 
be  highly  variable  and  the  govern¬ 
ment,  unlike  the  commercial  sector, 
may  require  a  contractor  to  certify  the 
information,  which  can  introduce  le¬ 
gal  consequences  should  the  vendor 
misrepresent  its  costs.  While  some 
large  commercial  companies  may  re¬ 
quire  certified  cost  data,  the  only  po¬ 
tential  penalty  from  misrepresentation 
is  that  the  buyer  may  take  his  or  her 
future  business  elsewhere. 

2.  These  sites  as  well  as  the  specific  en¬ 
gine  will  not  be  identified.  A  precon¬ 
dition  for  gaining  access  to  some  of 
this  data  was  that  the  manufacturer 
would  not  be  identified  so  that  their 
proprietary  information,  the  discount 
their  best  commercial  customer  pays, 
would  not  be  publicly  revealed,  par¬ 
ticularly  to  their  other  customers. 
Confidentiality  is  a  common  feature 
in  DoD  acquisition  research  and  can 
be  found  in  seminal  works  by  such 
distinguished  researchers  as  Peck  and 
Scherer  (1962)  and  Gansler  (1978, 
1982). 

3.  A  BPA  is  not  a  contract;  rather,  it  is 
more  of  a  charge  account  with 
qualified  source(s)  of  supply  to  fill. 


4.  The  field  engineering  support  for  $3 
million  was  provided  to  the  field  in 
anticipation  of  problems.  Adding  15 
percent  to  the  first  $20  million  in  spare 
part  purchases  was  viewed  as  an  effi¬ 
cient  means  of  collecting  the  money 
and  paying  for  services  that  were  typi¬ 
cally  difficult  and  time  consuming  to 
contract  for.  No  longer  did  the  item 
manager  have  to  contract  individually 
when  a  crisis  or  problem  developed 
in  the  field.  He  could  now  immedi¬ 
ately  deploy  contractor  resources  to 
fix  problems.  In  effect,  contracting 
velocity  (speed  at  which  a  contract  is 
put  in  force)  is  significantly  increased 
and  customer  wait  time  lowered.  This 
is  similar  to  contracts  used  by  some 
airlines.  Besselman  (1998)  identified 
one  major  commercial  airline  that 
buys  parts  and  services  for  at  least  one 
engine  in  this  manner. 

5.  Although  the  item  manager,  buyer, 
and  contracting  officer  could  not 
explain  this  discrepancy,  it  does  not 
materially  change  the  final  results. 

6.  In  the  aggregate,  the  DoD  is  the  larg¬ 
est  buyer,  but  the  majority  of  part  sales 
go  to  non-DoD  buyers.  The  volume 
of  purchasing  by  the  largest  commer¬ 
cial  customer  is  comparable  to  one  of 
the  DoD’s  individual  depots. 
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7.  Some  contractors  will  go  to  great 
lengths  to  have  an  item  declared  com¬ 
mercial.  During  a  fall  1999  visit  to  a 
depot,  one  reputable  firm  was  at¬ 
tempting  to  use  sales  of  military  parts 
to  mercenaries  on  the  African  conti¬ 
nent  as  evidence  of  “commercial 
sales.” 

8.  One  of  today's  widely  held  urban  leg¬ 
ends  is  that  there  is  one  way  the 
commercial  sector  does  something. 
Purchasing  is  no  exception.  We  have 
seen  suppliers  to  a  large  retailer  and 
fast  food  chain  provide  cost  visibility 
into  such  commercial  items  as  baby 
clothing  and  ground  beef  and  other 
ingredients,  respectively. 

9.  Monopsony,  the  case  of  only  one  cus¬ 
tomer,  is  analogous  to  monopoly. 
Thus,  in  a  monopsony,  the  buyer  can 
usually  do  better  by  gathering  infor¬ 
mation  about  the  cost  of  production, 
but  only  by  introducing  inefficiencies 
(e.g.,  see  Blair  and  Harrison,  1993), 
We  suggest  using  price-based  pur¬ 
chasing  even  when  DoD  is  the  only 
customer  as  long  as  many  firms 
supply  the  item  under  competitive 
conditions. 


10.  This  technique  is  nearly  identical  to 
an  approach  DLA  pioneered  with 
Honeywell  and  showcased  at  the  2000 
Acquisition  and  Logistics  Reform 
Week,  Washington,  DC,  May  22-26, 
2000. 

Transaction  costs  are  typically  costs 
of  contracting  with  third  parties  (e.g., 
see  Williamson,  1985).  In  this  case, 
these  costs  include  the  cost  of  collect¬ 
ing  and  analyzing  cost  data  and 
negotiating  contracts. 
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OPINION 


BUILDING  A 
BUSINESS  CASE  FOR 
MODELING  AND  SIMULATION 


C.  David  Brown,  Ph.D.,  COl  Gordon  Grant,  Canadian  Fortes, 
ITC  Donald  Kokhman,  USA,  COl  Robert  Reyenga,  USA, 
and  if  Col  Terente  Szanto,  USAF 

Modeling  and  simulation  technology  is  the  use  of  models  to  develop  data  as  a 
basis  for  making  managerial  or  technical  decisions.  It  can  be  a  valuable  tool  for 
program  managers— but  it  is  one  that  is  vastly  under-used.  This  article  provides 
a  business-case  framework  (a  methodology  to  evaluate  investment  opportu¬ 
nities)  for  program  managers  within  the  Department  of  Defense  to  use  when 
determining  how  to  apply  modeling  and  simulation  in  project  management. 


The  use  of  modeling  and  simulation 
(M&S)  is  widely  misunderstood 
within  the  Department  of  Defense 
(DoD).  M&S  is  the  use  of  models,  either 
statically  or  over  time,  to  develop  data  as 
a  basis  for  making  managerial  or  techni¬ 
cal  decisions  (DoD,  1997).  Models  are 
physical,  mathematical,  or  logical  repre¬ 
sentations  of  a  system,  entity,  phenom¬ 
enon,  or  process.  Simulations  are  meth¬ 
ods  for  implementing  models  over  time. 
Normally,  we  associate  simulations  with 
a  software  program  that  implements  mod¬ 
els  over  time,  within  the  context  of  a  given 
scenario  (Defense  Modeling  and  Simula¬ 
tion  Office,  1996).  Simulations  permit  the 
user  to  assess  variables  and  the  predict¬ 
ability  of  a  single  or  series  of  outcomes. 


Nowhere  is  the  misunderstanding  more 
painfully  obvious  than  within  the  program 
management  offices  of  the  DoD.  Some 
program  managers  believe  M&S  is  para¬ 
mount  to  effective  project  development 
and  place  the  requisite  investment  in  it 
(and  this  article  highlights  examples  of 
some  such  programs).  But  many  program 
managers  remain  both  skeptical  and  sus¬ 
picious.  Recent  government  direction  to 
use  simulation-based  acquisition  in  DoD 
programs  is  an  example  of  a  policy  with 
good  intentions  but  poorly  shaped  execu¬ 
tion.  This  edict  has  been  met  with,  at  best, 
marginal  acceptance,  and  at  worst,  abject 
resentment. 

Such  resentment  and  apprehension 
spring  from  institutionalized  biases, 
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including  DoD  funding  procedures,  that 
work  against  optimizing  the  potential 
gains  of  employing  modeling  and  simu¬ 
lation.  By  far,  the  severest  criticisms  tar¬ 
geted  at  M&S  center  on  the  debate  over 
return  on  investment  (ROI).  As  we  will 
discuss  later,  ROI  is  just  one  of  many  tech¬ 
niques  in  evaluating  the  use  of  M&S  in  a 
program.  Ostensibly,  DoD  has  accepted 
this  new  technology  as  a  means  of  reduc¬ 
ing  costs,  increasing  cost  avoidance,  and 
banking  the  residual  benefits  for  other 
projects. 

Many  program  managers  argue  that  the 
entire  acquisition  system  is  focused  on 
getting  a  project  into  production,  through 
performance  trials,  and  permanently  into 
the  military’s  inventories.  Seldom  are  they 
. . . .  . - . , :  given  suffi¬ 
cient  funds, 
staff,  or  time  to 
investigate  the 
potential  ben¬ 
efits  of  tools  or 
technologies 
such  as  M&S. 
Importantly, 
leadership  pro¬ 
vides  little  in¬ 
centive  to  capture  data,  build  expensive 
models,  or  conduct  additional  analyses  to 
transfer  M&S  results  to  other  projects. 
Simply  put,  program  managers  are  under 
intense  pressure  to  complete  their  pro¬ 
grams  on  or  under  budget  and  within  time¬ 
lines.  Existing  programs  lack  enticement 
to  develop  new  models  or  simulation  tools 
that  may  have  wider  application  to  other 
programs,  or  that  will  be  much  cheaper  to 
operate  and  sustain.  With  few  exceptions, 
these  occurrences  were  more  a  result  of 
coincidence  than  deliberateness. 


"Simply  put, 
program  managers 
are  under  intense 
pressure  to 
complete  their 
programs  on  or 
under  budget  and 
within  timelines." 


Finally,  perhaps  the  greatest  impedi¬ 
ment  to  M&S  acceptance  is  lack  of  knowl¬ 
edge.  Many  people  do  not  understand  the 
potential  benefits  of  this  technology,  or 
how  to  define  needs  and  produce  the  right 
tools  that  will  help  the  project.  Unfortu¬ 
nately,  this  reticence  may  be  reinforced 
by  an  institution  which  neither  favors  nor 
rewards  risk  takers.  Sometimes,  a  program 
manager  may  not  know  for  certain  if  a 
major  investment  in  M&S  is  warranted. 
If  a  program  manager  wants  to  invest  in 
the  technology  “just  to  see  if  there  is  a 
benefit,”  he  or  she  faces  criticism  if  the 
results  are  not  positive.  Thus,  a  program 
manager  must  weigh  the  costs  of  the  risk, 
and  consequently  few  take  the  chance, 
preferring  more  traditional  approaches — 
the  building  of  expensive  mock  ups;  the 
use  of  labor-,  time-,  and  money-intensive 
trials,  and  incurred  costs  of  waste. 

The  deputy  project  manager  (DPM)  of 
the  Joint  Strike  Fighter  (JSF)  praised  the 
future  of  M&S  while  lamenting  the  mili¬ 
tary’s  reticence  to  embrace  it.  “The  Joint 
Strike  Fighter  Project  achieved  immeasur¬ 
able  benefits  from  its  innovative  use  of 
modeling  and  simulation.  More  important, 
I  was  impressed  with  the  tremendous  in¬ 
tellect  and  drive  of  many  of  the  staff  who 
were  willing  to  try  new  models.  They  did 
simulation  runs — sometimes  up  to  1,000, 
in  order  to  bring  this  project  along  a  de¬ 
velopment  path  to  bring  the  necessary 
technology-enhanced  fighter  into  the  21st 
century.  They  took  the  risks  and  it’s  pay¬ 
ing  off”  (1999).1  The  DPM  made  this  blunt 
statement — “No  high-tech  project  will 
achieve  any  significant  success  if  it  does 
not  incorporate  M&S.  However,  we  in 
DoD  just  aren’t  ready  to  capitalize  on  this 
technology.” 


312 


Building  a  Business  Case  for  Modeling  and  Simulation 


Joint  Strike  Fighter 


While  all  of  this  bodes  well  for  an 
anecdotal  argument  supporting  M&S, 
what’s  needed  is  a  more  reasoned  and 
defensible  process  for  determining  M&S 
investments.  While  few  refute  the  intui¬ 
tive  benefits  of  M&S,  program  managers 
quite  rightly  argue  that  any  tool  must  be 
first  measured  against  its  potential  ben¬ 
efits  before  it  is  used.  For  example,  given 
that  Boeing  spent  $2  billion  on  M&S  for 
its  111  airplane,  are  comparable  levels  of 
investment  affordable  to  program  manag¬ 
ers  of  programs  of  similar  magnitude? 
Obviously,  the  question  is  not  easily  an¬ 
swered.  It  depends  on  a  variety  of  factors, 
including  the  project’s  funding,  period  to 
recoup  the  investment,  and  perceived  ben¬ 
efits  to  developing  and  using  M&S  in  the 
program.  The  crux  of  the  problem  is  the 
dilemma  of  costs  versus  risks  and  the 
potential  return.  Program  managers  need 


a  methodology  to  evaluate  investment 
opportunities. 

Methodoiogy  ahd  Approach _ 

The  authors  reviewed  selected  M&S 
efforts  within  civilian  industry  and  DoD. 
We  visited  several  key  manufacturing  and 
service  industries,  which  were  also  wres¬ 
tling  with  the  same  subject — the  assess¬ 
ment  of  M&S  investment.  Understand¬ 
ably,  each  sector  had  a  slightly  different 
motivation  or  incentive  to  invest  in  M&S. 
One  company  was  concerned  about  long¬ 
term  applicability  and  the  transference  of 
technology  to  future  programs.  Another 
was  in  the  business  of  capturing  data  and 
shaping  models  and  the  requisite  simula¬ 
tion  runs  to  satisfy  client  needs.  They  were 
using  training  models  and  virtual  reality 
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technology  as  a  means  to  reduce  training 
costs  and  time.  Another  believes  a  judi¬ 
cious  application  of  M&S  reduces  direct 
manufacturing  costs.  For  example,  on  the 
JSF  program,  simulations  are  improving 
mechanical  tolerances  such  that  develop¬ 
ers  project  shim  stock  weight  reduction 
from  an  average  of  40  pounds  per  aircraft 
(as  is  the  case  with  the  F-16)  down  to  less 
than  1  pound.  Such  projections  are  rea¬ 
sonable,  based  on  actual  data  from  Boeing 
111  design  and  projection. 

While  there  are  some  common  appli¬ 
cations  industry-wide,  DoD  is  not  focused 
on  profit,  but  on  performance  and  total 
ownership  costs.2  We  surveyed  a  large 
number  of  DoD  program  managers,  with 
programs  in  various  stages  of  the  acqui¬ 
sition  process.  The  survey  requested  gen¬ 
eral  information  from  the  program  man¬ 
agers  regarding  how  they  made  decisions 
on  M&S  investments.  Additionally,  in  an 
effort  to  maintain  balance,  we  also  sur¬ 
veyed  several  government  contractors.  We 


also  conducted  a  literary  search  to  cap¬ 
ture  the  body  of  knowledge  in  non-govem- 
ment  organizations  related  to  justifying 
M&S  investments. 

Results  of  Surveys 

In  general,  survey  responses  indicated 
that  program  managers  are  investing  in 
M&S  to  support  program  development; 
however,  most  of  the  investment  decisions 
were  based  on  intuition  or  need-based  fac¬ 
tors.  Most  decisions  were  made  without 
detailed  quantitative  analysis.  Ostensibly, 
program  managers  accepted  modest 
investment  in  M&S  because  they  believed 
that  it  would  benefit  the  program.  These 
efforts,  however  commendable,  lacked  a 
methodical  cost-benefit  analysis.  Addi¬ 
tionally,  the  lack  of  a  structured  business 
case  analysis  made  it  difficult,  if  not  im¬ 
possible,  for  program  managers  to  articu¬ 
late  or  substantiate  their  investment  strat¬ 
egy.  In  some  cases  it  came  down  to  a  pro¬ 
gram  manager  wishing  to  explore  M&S 
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without  knowing  what  benefits  might  be 
achieved.  This  is  not  really  taking  a  risk; 
it  is  more  an  exploratory  probe  into  a  new 
field.  These  program  managers  were  em¬ 
bracing  new  technology — but  without  a 
sound  business  case  approach,  they  could 
not  really  assess  if  the  investment  would 
realize  considerable  benefits  or  generate 
prohibitive  costs. 

Most  program  managers  justified  their 
M&S  investment  based  on  one  or  more 
of  the  following: 

•  reducing  design  cycle  time; 

•  augmenting  or  replacing  physical  tests; 

•  helping  resolve  limitations  of  funds, 

assets  or  schedules;  or 

•  providing  insight  into  issues  that  were 

impossible  or  impracticable  to  examine 

in  other  ways. 

Much  of  the  feedback  reflected  that 
program  managers  had  tried  to  examine 
costs  and  measure  them  against  bene¬ 
fits — but  without  the  help  of  any  business 
case  analysis  format.  Consequently,  ap¬ 
proaches  and  results  varied.  Few  reported 
to  have  used  a  disciplined  approach.  The 
use  of  inconsistent  applications  or  ap¬ 
proaches  leads  to  mixed  results  that  cannot 
be  readily  compared  or  evaluated.  The 
majority  of  respondents  suggested  the 
question  of  using  M&S  was  not  one  of 
“Should  I”  but  rather,  “How  can  I?”  This 
demonstrates  general  acceptance  that 
M&S  is  required  in  an  efficiently  man¬ 
aged  program.  But  it  begs  the  question  “at 
what  cost?”  Indeed,  the  cost  of  investment 
may  be  prohibitively  expensive,  or  there 
may  be  only  the  most  marginal  long-term 


benefits  to  the  department.  In  such  cases, 
the  decision  maker  should  select  from  al¬ 
ternatives,  be  it  foregoing  M&S,  partner¬ 
ship  with  other  program  managers  to  share 
costs,  or  leveraging  the  investment  of  oth¬ 
ers.  This  is  not  a  moot  point,  given  DoD’s 
fiscal  limitations. 

Program  managers  want  confirmation 
that  investment  in  M&S  will  yield  direct 
savings  within  their  budgets.  Considering 
that  much  of  the  benefit  of  M&S  invest¬ 
ment  is  intangible,  traditional  measure¬ 
ment  approaches  may  not  provide  an 
accurate  assess¬ 


ment.  Addition¬ 
ally,  program 
managers  may 
be  understand¬ 
ably  too  paro¬ 
chial  in  that,  if 
the  return  is  not 
significant  for 
their  immediate 


"Program 
managers  want 
confirmation  that 
investment  in  M&S 
will  yield  direct 
savings  within 
their  budgets." 


project  needs,  they  may  dismiss  these 
tools.  But  projects  are  relatively  short¬ 
lived — we  may  very  well  be  missing  some 
of  the  longer-term  residual  benefits.  While 
traditionally  the  benefits  of  M&S  tend  to 
be  discussed  in  terms^of  return  on  invest¬ 


ment,  several  alternatives  for  business  case 
analysis  can  just  as  effectively  justify 
M&S  investments.  The  challenge  is  to 
define  an  appropriate  strategy  and 
priorities  to  address  the  business  value 
proposition. 

A  disciplined  approach  and  methodol¬ 
ogy  has  many  benefits.  It  can  help  bring 
the  aggregate  benefits  into  focus  and 
strengthen  the  argument  for  M&S  invest¬ 
ment.  A  business  case  analysis  provides  a 
convenient  mechanism  for  project  man¬ 
agement.  It  can  be  an  easy-to-follow,  logi¬ 
cal  thread.  It  also  lays  the  groundwork  for 
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others  to  attain  information  that  will  help 
their  respective  programs.  This  approach 
forces  a  timeline,  captures  benefits,  and 
enables  authorities  to  decide  if  the  return 
is  worth  the  pursuit. 

Making  the  Business  Case 


Corporate  America  is  taking  a  methodi¬ 
cal  approach  to  investment  decisions 
regarding  M&S.  Although  they  are  prima¬ 
rily  profit  driven,  they  share  common 
maxims  of  production — such  as  cost 
reduction,  efficiency,  and  cost  avoid¬ 
ance — to  that  of  DoD.  Essentially,  a  busi¬ 
ness  case  will  assist  the  program  manager 
in  evaluating  which  of  a  number  of  logi¬ 
cal  packages  of  alternatives  will  best  meet 
the  program’s  objective. 

Existing  Guidelines  and  Instructions 

The  survey  revealed  that  although  DoD 
has  issued  guidance  on  this  subject,  few 

program  man¬ 
agers  reported 

"The  baseline  that  they  were 

provides  a  bench-  following  it. 

mark  from  which  DoD  guidance 

decisions  will  be  on  investment 

weighed  and  guidance  is 

assessed."  provided  by 

DoD  Instruc¬ 
tion  (DoDI) 
Number  7041.3,  Economic  Analysis  for 
Decisionmaking,  (1995).  Enclosure  3  of 
the  DoDI,  Procedures  for  Economic 
Analysis ,  provides  an  insightful  overview 
of  methodology,  criteria  and  a  discussion 
of  sensitivity  analysis  (1995).  The  Gen¬ 
eral  Services  Administration’s  Informa¬ 
tion  Technology  Capital  Planning  and 
Investment  Guide ,  (1998),  also  provides 


procedures  for  economic  analysis.  Some 
services  may  also  have  supplemental  guid¬ 
ance  (U.S.  Army  cost  and  Economic 
Analysis  Center,  1995). 

A  Business  Case  Framework _ 

Given  our  research,  the  results  of  sur¬ 
veys,  discussions  with  industry  and  our 
literature  search,  we  recommend  a  seven- 
step  process  for  assessing  the  utility  of 
M&S  investments: 

•  Establish  a  baseline. 

•  Establish  a  vision  and  direction. 

•  Quantify  the  costs  and  benefits  of 
altematives/capabilities. 

•  Evaluate  alternatives. 

•  Conduct  sensitivity  analysis. 

•  Develop  a  migration  strategy. 

•  Monitor  the  process  and  continue  to 
assess  results  through  formalized 
feedback. 

Step  1.  Establishing  a  Baseline 

As  with  many  other  decision-making 
processes,  the  first  step  is  to  establish  an 
accurate  baseline.  The  baseline  provides 
a  benchmark  from  which  decisions  will 
be  weighed  and  assessed.  The  baseline 
should  include  a  clear  enunciation  of 
assumptions  and  constraints.  Assumptions 
are  explicit  statements  describing  the 
present  and  future  environment.  They  re¬ 
duce  complex  situations  into  manageable 
proportions.  These  assumptions  normally 
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provide  some  comment  on  the  estimated 
future  workload,  the  useful  life  of  the  in¬ 
vestment  or  system,  and  the  period  of  time 
over  which  alternatives  will  be  compared. 
Assumptions  should  also  discuss  sunk 
costs  and  realized  benefits,  but  are  not 
included  as  part  of  the  baseline.  Con¬ 
straints  are  those  factors  that  limit  alter¬ 
natives.  Normally  they  are  expressed  in 
terms  of  time,  finances,  institutional  or 
regulatory  statutes,  or  directives  and 
physical  plant  and  assets. 

The  baseline  must  identify  the  “higher- 
value”  portions  of  a  program  in  order  to 
evaluate  the  appropriateness  of  various  al¬ 
ternatives.  For  example,  in  the  case  of  an 
aircraft,  75  percent  of  program  hours 
might  be  expended  on  the  air  vehicle  team. 
On  closer  examination,  one  might  find  that 
45  percent  of  that  time  goes  into  airframe 
design,  and  of  that  figure,  90  percent  is 
expended  on  mid-fuselage  development. 
With  this  informed  examination,  the  pro¬ 
gram  manager  will  be  better  able  to  allo¬ 
cate  M&S  spending  where  it  will  offer  the 
greatest  potential  savings  or  benefits.  The 
baseline  must  determine  these  high  value 
areas  for  effective  business  case  analysis. 
These  program  specifics  form  the  drivers 
to  the  program,  which  in  turn  drive  the 
investment  process. 

Step  2.  Establishing  Vision  and  Direction 

One  must  look  to  the  future,  then  bridge 
the  gap  between  present  knowledge  and 
that  required  to  make  future  products  a 
reality.  A  program  manager  must  estab¬ 
lish  the  program’s  vision.  As  with  other 
technologies,  the  program  manager’s 
vision  should  consider  how  M&S  tools 
can  improve  program  costs,  scheduling, 
and  performance,  and  whether  scientific 
knowledge  exists  to  support  such  M&S 


investment.  The  vision  should  drive  what 
the  M&S  tools  should  be  trying  to  solve, 
not  the  other  way  around. 

For  example,  although  the  Big  Three 
auto  companies  in  Detroit  are  producing 
very  similar  products,  each  is  using  M&S 
in  very  different  ways,  based  upon  their 
strategic  visions.  One  is  concentrating 
heavily  on  using  M&S  in  design  at  a  single 
location;  another  is  concentrating  on  mov¬ 
ing  large  amounts  of  digital  information 
around  the  world  in  order  to  develop  a  glo¬ 
bal  engineering  capability;  and  the  third 
is  heavily  in¬ 
vesting  in  re¬ 
ducing  the  costs 
of  manufactur¬ 
ing.  Of  course 
each  is  doing 
some  or  all  of 
these,  but  the 
vision  provides  the  focus  for  allocating 
their  scarce  resources.  It  clearly  identifies 
what  is  to  be  achieved,  without  dictating 
how  it  will  be  done. 

Step  3.  Quantify  the  Costs  and  Benefits 
of  Alternatives  and  Capabilities 

Alternatives  are  logical  packages  of 
initiatives  that  work  well  together  (Kid- 
well,  1998).  One  alternative  is  the  status 
quo — that  is,  what  we  identified  as  the 
baseline  in  Step  1.  In  some  cases  the  base¬ 
line,  often  a  physical  test,  can  be  more  cost 
effective  than  the  use  of  M&S.  Other 
alternatives  should  represent  various  com¬ 
binations  of  M&S  tools  that  help  achieve 
the  vision.  In  determining  alternatives,  one 
must  consider  both  immediate  and  long¬ 
term  effects.  First,  what  is  it  the  program 
needs  to  perform  better?  Second,  what 
does  the  program  need  in  order  to  survive 
until  the  next  stage?  Most  certainly,  a 


I 
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program  must  satisfy  a  requirement,  but 
it  must  also  endure  and  survive  each  step 
of  the  process.  It  does  no  good  for  a  pro¬ 
gram  to  bankrupt  itself  with  massive  un¬ 
focused  M&S  investment  early  on.  Each 
investment  must  produce  value  during  a 
time-frame  that  is  appropriate  for  the  pro¬ 
gram.  Reality  requires  meeting  near-term 
milestones.  The  alternatives  must  also 
consider  the  technological  advances  in 
M&S  tools  that  might  occur.  A  program 
might  not  be  justified  in  spending  huge 
sums  of  money  on  M&S  technology  that 
will  be  superceded  and  rendered  obsolete 
in  2-5  years. 

The  program  manager  must  identify  all 
costs  that  are  incident  to  achieving  each 
alternative.  Models  and  simulations  can 
be  expensive  to  develop,  particularly  in 
domains  where  the  scientific  principles  are 
not  fully  understood  as  applied  to  the 
problem.  While  additional  research  can  fill 
the  knowledge  voids,  the  cost  of  this 
research  must  be  factored  into  the  analy¬ 
sis  of  alternatives.  These  should  include 


"Identifying 
alternatives  may 
be  the  most 
difficult  portion 
of  the  process." 


the  opportu¬ 
nity  costs  of 
assets  and  re¬ 
sources,  which 
are  the  alterna¬ 
tive  value  fore¬ 
gone  when  an 
asset  is  used 


for  other  purposes  (DoDI,  1995).-  They 
also  include  nonrecurring  and  recurring 
costs.  Life-cycle  costs  should  include  all 
costs,  nonrecurring  and  recurring,  that 
occur  over  the  life  of  an  alternative  (Gen¬ 
eral  Services  Administration,  1998). 

Identifying  alternatives  may  be  the 
most  difficult  portion  of  the  process.  Ben¬ 
efits  must  be  viewed  primarily  in  terms 
of  measurable  value.  Expected  benefits 


should  flow  from  the  clear  operating  vi¬ 
sion  developed  in  Step  2.  Enigmatically, 
there  are  both  quantifiable  and  unquan- 
tifiable  benefits.  The  former  have  some 
tangible  or  readily  identified  returns;  the 
latter  have  less  so.  Additionally,  there  may 
be  benefits  that  have  no  intrinsic  value  to 
one  program  but  provide  value  to  others. 
We  call  these  external  benefits. 

Quantifiable  benefits.  These  include 
cost  savings,  time  improvement,  accelera¬ 
tion  of  deliverables,  quality  enhancement 
and,  in  most  cases,  cost  avoidance  that  is 
directly  related  to  the  program.  The  alter¬ 
natives  must  also  consider  existing  sys¬ 
tems  and  programs.  If  we  are  to  measure 
improvements  from  an  “as-is  baseline” — 
we  need  not  start  from  ground  zero.  It  may 
be  possible  for  program  managers  to  look 
at  M&S  initiatives  in  other  programs, 
assess  their  applicability,  and  leverage 
them  for  success.  Cost  associated  with 
these  alternatives  should  be  less,  given  that 
a  majority  of  the  investment  would  be  a 
sunk  cost  borne  by  others.  Similarly,  pro¬ 
gram  managers  must  consider  whether 
partnering  with  another  program,  thereby 
sharing  costs,  is  a  possible  alternative  to 
reduce  up-front  investment.  Program  man¬ 
agers  should  ensure  they  have  examined 
all  potential  benefits  by  using  published 
references  and  experts  in  the  field  of  cost 
analysis. 

Unquantifiable  benefits.  Traditionally, 
we  have  considered  the  issues  of  risk 
reduction,  organizational  efficiency,  tech¬ 
nology  transference,  product  safety,  and 
environmental  impact  reductions  as  un¬ 
measurable  and  therefore  unquantifiable. 
However,  these  are  important  issues,  and 
program  managers  must  consider  them  in 
their  analysis.  To  illustrate  this  point,  we 
will  address  technology  transference. 
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The  Grizzly 


M&S  technology  transference  can  sig¬ 
nificantly  influence  costs,  but  in  today’s 
DoD  environment  it  has  yet  to  receive 
adequate  attention.  Given  the  shrinking 
public  purse  and  the  demand  for  greater 
accountability  and  responsibility  for  the 
dispersal  of  funds,  all  program  managers 
must  show  due  diligence  in  their  public 
spending.  They  must  consider  the  residual 
benefits  of  technology  transference. 

Some  M&S  investment  might  be  of  use 
to  other  projects  and  program  managers. 
For  example,  the  Grizzly3  program  man¬ 
ager  invested  heavily  in  chassis  M&S  to 
support  short-term  design  and  perfor¬ 
mance  analysis.  This  M&S  investment  re¬ 
sulted  in  $21  million  worth  of  quantifi¬ 
able  benefits  to  the  Grizzly  program.  The 
Grizzly  program  manager  funded  the 
M&S  effort  through  internal  reallocation 
of  funds.  The  program  manager’s  supervi¬ 
sor,  program  manager  Combat  Mobility 


Systems,  recognized  the  potential  for  the 
use  of  these  models  for  both  other  pro¬ 
gram  requirements  and  in  other  programs 
sharing  the  Grizzly’s  chassis.  Program 
manager  Combat  Mobility  Systems  lever¬ 
aged  the  Grizzly  Program’s  M&S  invest¬ 
ment,  securing  funding  to  expand  the  ap¬ 
plicability  of  the  initial  investment  into 
other  programs  and  to  support  other  long¬ 
term  Grizzly  requirements. 

When  forecasting  near-term  savings  in 
design  and  production  costs,  one  of  our 
surveyed  companies  accrued  substantial 
unquantifiable  benefits.  The  engineers 
made  a  substantial  leap  in  M&S  knowl¬ 
edge  when  learning  how  to  define  data 
needs,  how  to  shape  models,  and  how  to 
refine  simulation  runs,  to  narrow  the  band¬ 
width  of  problem  solving.  The  resulting 
expertise,  data,  and  process  could  be  ap¬ 
plied  to  future  projects.  Not  surprisingly, 
the  company’s  models  and  data  bank  are 
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the  envy  of  the  industry.  This  was  an 
unquantifiable  gain.  Another  unquanti- 
fiable — or  at  least  indiscrete — benefit  of 
M&S  is  the  competitive  advantage  it  pro¬ 
vides.  This  relates  back  to  establishing  a 
clear  vision  so  that  early  investments  will 
lead  the  company  to  where  it  wants  to  be 
in  the  future. 


Unfortunately,  many  program  man¬ 
agers  dismiss  the  concept  of  unquanti¬ 
fiable  benefits,  program  managers  rarely 
track  these  benefits,  or  those  outside  the 
program’s  realm  with  any  real  vigor.  They 
don’t  afford  them  reasonable  weight  when 
analyzing  costs  and  their  alternatives.  Simi¬ 
larly,  external  benefits  may  exist,  not  only 
to  service  DoD  at  large,  but  to  external 
agencies  and  businesses. 

External  benefits.  These  are  benefits 
which  do  not  bring  direct  return  or  sav¬ 
ings  to  the  unique  program  being  man¬ 
aged,  but  have  applicability  beyond  the 

program  man¬ 
ager’s  purse. 

"Vie  must  As  mentioned 


overcome  the 
institutional  bias 
that  forces  program 
managers  to  ignore 
external  benefits." 


previously, 
many  M&S  in¬ 
itiatives  and 
their  products 
can  either  be 


modified  or  di¬ 


rectly  transferred  to  other  programs. 
Again,  looking  at  the  Grizzly  program,  the 
contractor  supporting  the  program  man¬ 
ager  (United  Defense  Limited  Partner¬ 
ship)  developed  a  common  product  model 
database  that  benefited  efforts  at  Aberdeen 


Proving  Ground,  The  Army  Warfighting 
Analysis  and  Integration  Center,  Water¬ 
ways  Experiment  Station  and  National 
Training  Center  projects.  Thus  there  is  a 
residual  savings  for  follow-on  users.  The 
surveyed  program  managers  did  identify 


a  problem  with  the  high  cost  of  collecting 
data  and  maintaining  the  database.  While 
the  cost  of  performing  this  might  be  high, 
or  perhaps  even  prohibitive  to  one  project, 
it  could  be  cost  effective  to  several  other 
end  users.  We  must  overcome  the  institu¬ 
tional  bias  that  forces  program  managers 
to  ignore  external  benefits.  A  program 
manager  has  no  incentive  to  take  on  an 
M&S  investment  unless  he  or  she  can  jus¬ 
tify  the  expense  from  the  existing  (and 
often  cash-strapped)  program.  The  follow¬ 
ing  example  demonstrates  how  external 
benefits  can  show  marked  savings  to  DoD 
and  the  public. 

For  years  the  Aberdeen  Test  Center 
(ATC)  put  vehicles  through  multiple  runs 
over  ground  to  determine  wear  and  tear 
on  parts  and  the  resultant  performance 
degradation.  This  required  a  large  num¬ 
ber  of  personnel,  vehicles  and  time  to  log 
thousands  of  miles  to  achieve  statistically 
significant  results.  Since  then,  the  ATC 
completed  an  intensive  project  where  data 
was  collected  describing  the  complete  pro¬ 
file  of  the  course.  Subsequently,  engineers 
built  the  models  and  now  conduct  or  aug¬ 
ment  many  of  these  tests  on  a  virtual  prov¬ 
ing  ground  using  simulations  in  lieu  of 
hardware.  The  simulations  are  so  accurate 
that  they  have  been  able  to  document  mil¬ 
lions  of  dollars  in  cost  avoidance  for  test¬ 
ing  of  Army  programs,  while  concomi¬ 
tantly  helping  the  Army  make  the  requi¬ 
site  decisions  for  product  and  performance 
improvements.  Not  surprisingly,  others 
outside  the  DoD,  including  private  indus¬ 
try,  insurance  corporations,  and  the  De¬ 
partment  of  Transportation,  also  want  to 
use  this  product.  (According  to  DoDI 
7041.3,  societal  costs  and  benefits  outside 
the  federal  government  are  usually  not 
included  in  a  DoD  analysis). 
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Although  this  program  has  universal 
application,  ATC  had  to  offset  the  costs 
of  this  M&S  project  through  internal  sav¬ 
ings  in  manpower  and  overhead,  rather 
than  being  permitted  to  share  the  cost  with 
other  programs.  An  alternative  strategy 
would  be  to  identify  potential  users  in 
advance  and  share  the  developmental 
costs.  Undeniably,  program  managers 
should  take  the  first  step  to  accrue  direct 
benefits  to  their  programs.  But  they  con¬ 
tinue  to  bypass  transferable  benefits  sim¬ 
ply  because  direct  program  constraints 
preclude  further  investment  of  resources. 

Perhaps  the  real  value  of  identifying 
quantifiable  and  unquantifiable  benefits  is 
in  helping  others  outside  the  program  to 
realize  potential  synergies  of  reuse.  For 
example,  the  program  executive  officer, 
who  is  charged  with  program  oversight, 
will  have  better  visibility  into  require¬ 
ments  and  the  potential  benefits.  He  or  she 
can  more  accurately  assess  M&S  invest¬ 
ment  in  relation  to  a  broader  sphere  of 
programs.  Operational  analysis  and  train¬ 
ing  are  just  a  few  examples.  Many  of  these 
benefits,  while  external  to  an  individual 
program  manager,  may  be  internal  benefit 
from  the  program  executive  officer’s  per¬ 
spective.  Armed  with  this  information,  a 
program  executive  officer  may  choose  to 
redirect  funding  from  other  sources  into 
the  program,  and/or  direct  a  program  man¬ 
ager  to  take  a  course  of  action  which  may 
not  be  cost-effective  in  a  micro  perspec¬ 
tive;  but  will  bring  an  aggregate  gain  that 
far  outweighs  the  individual  investment. 
But  sound  management  is  predicated  on 
program  managers  providing  the  program 
executive  officer  with  data  and  informa¬ 
tion  drawn  from  the  program  manager’s 
business  case  analysis. 


Step  4.  Evaluate  Alternatives 

We  must  compare  the  costs  and  ben¬ 
efits  of  each  alternative  and  rank  them. 
Such  comparisons  must  be  accomplished 
using  both  quantitative  and  qualitative 
techniques  and  criteria.  Quantitative  tech¬ 
niques  include  net  present  value,  benefit 
cost  ratio,  return  on  investment,  payback 
method,  internal  rate  of  return,  hurdle  rate, 
and  cost  effectiveness  analysis.4  Qualita¬ 
tive  evaluation 
considerations 

"Undeniably, 
program  managers 
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such  as  relation¬ 
ship  to  business 
strategy,  sched¬ 
ule  risk,  organi-  S 
zational  and 
technical  risks,  | 
social  benefits, 
and  legal  and  regulatory  requirements  may 
greatly  alter  the  quantitative  ranking. 

The  choice  of  appropriate  tools  is 
program  and  situation  dependent,  and  can 
greatly  influence  the  outcome  of  the 
analysis.  These  tools  will  aid  decision 
makers  in  accurately  evaluating  all  alter¬ 
natives  such  that  all  costs  and  benefits  are 
viewed  on  a  level  playing  field.  In  gen¬ 
eral,  each  feasible  alternative,  life-cycle 
costs  and  benefits,  are  adjusted  using 
discount  factors  to  account  for  the  time 
value  of  money.  A  complete  analysis  prop¬ 
erly  relates  quantitative  and  qualitative 
factors.  Given  the  importance  of  these 
choices,  one  should  seek  expert  advice  and 
guidance  before  proceeding. 


■mt 


Step  5.  Conduct  Sensitivity  Analysis 

Sensitivity  analysis  is  an  essential  step 
in  the  decision  process,  as  it  accounts  for 
ever-present  uncertainties.  Such  analysis  re¬ 
peats  the  above  evaluation  of  alternatives 
with  changes  to  the  uncertain  variables 
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and  examines  the  effect  on  the  final  deci¬ 
sion.  The  outcome  will  provide  a  better 
understanding  of  the  robustness  of  the 
output. 

Sensitivity  analysis  is  highly  recom¬ 
mended,  even  if  there  appear  to  be  sig¬ 
nificant  differences  among  the  alterna¬ 
tives,  because  an  apparently  superior  so¬ 
lution  may  be  very  sensitive  to  changes 
in  a  single  variable.  Sensitivity  analysis 
is  required  when  differences  among  alter¬ 
natives  are  less  obvious  and  may  be  to¬ 
tally  driven  by 
variability  of 
"Program  key  input  fac- 

;  managers  face  tors-  The  key 

f  tremendous  factors  to  be 

I  pressure  to  bring  tested  may  in- 

a  product  into  use/f  elude,  but  are 

r  not  limited  to, 

i  project  or  pro¬ 

gram  length,  volume  or  quantity  and  mix 
of  production  units,  requirements,  con¬ 
figurations,  assumptions,  and  discount 
rates  and  other  economic  factors. 

Step  6.  Developing  a  Migration  Strategy 

After  determining  the  best  alternative, 
one  must  develop  a  sound  implementation 
plan  to  migrate  the  “winning  strategy”  into 
the  program.  This  plan  must  incorporate 
a  systematic  approach  whereby  the  devel¬ 
oper  plans  to  implement  the  identified 
drivers  and  capture  the  expected  benefits. 
Implementation  of  the  migration  strategy 
will  undoubtedly  force  changes  to  the 
program’s  plan  and  budget.  If  a  new  tool 
or  process  is  expected  to  save  money,  then 
those  savings  should  be  subtracted  from 
that  part  of  the  program  budget  and 
reassigned  elsewhere  as  an  up-front  action. 


Step  7.  Monitoring  and  Assessment 
through  Formalized  Feedback 

The  final  step  in  developing  a  business 
case  will  be  to  create  metrics  to  assess 
progress  toward  the  overall  vision.  These 
metrics  should  be  tied  to  the  changes  made 
to  the  program’s  acquisition  plan,  to  pro¬ 
vide  timely  feedback  on  their  success  in 
meeting  desired  results  in  performance, 
schedule,  and  cost.  Performance  metrics 
should  stem  from  the  needs  and  require¬ 
ments  that  alternatives  are  fulfilling,  and 
should  address  the  benefits  they  are  ex¬ 
pected  to  provide.  Schedule  and  cost 
metrics  must  also  be  developed  to  help 
ensure  programs  adhere  to  planned  costs 
and  schedules  (Kidwell,  1998;  DoD, 
1995;  GSA,  1998). 

Program  managers  face  tremendous 
pressure  to  bring  a  product  into  use.  While 
it  is  the  program  manager  who  can  best 
provide  monitoring  input,  funding  limits 
and  timelines  debilitate  an  aggregate  ap¬ 
proach  to  monitoring.  Unquestionably, 
program  managers  should  consider  the 
entire  life  cycle  of  the  project.  But  in  the 
present  acquisition  environment  there  is 
little  incentive  to  do  this.  Program  man¬ 
agers  are  the  lynchpin  to  success,  since 
they  hold  all  of  the  program-specific 
information. 

Program  managers  must  deliver  their 
programs  with  complementary  benefits 
first.  This  is  their  true  priority,  but  they 
should  also  identify  real  or  potential 
external  benefits  up  the  management  chain 
to  the  program  executive  officer.  That 
office  can  then  make  more  informed 
decisions  on  the  macro  benefits.  Program 
managers  should  consider  increasing  in¬ 
vestment  earlier  in  the  program  if  the  busi¬ 
ness  case  strongly  indicates  downstream 
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savings  as  a  result.  Program  executive 
officers  can  provide  the  attendant  over¬ 
sight  and  direction,  with  a  requisite  real- 
location  of  funds  when  it  is  in  DoD’s  best 
interest  to  do  so.  DoD  has  made  some  mar¬ 
ginal  progress,  as  historically,  program 
managers  did  not  worry  about  maintain¬ 
ability  and  sustainability  issues.  Now 
Gansler’s  revolution  in  military  affairs  is 
mandated  through  specifications  and  ex¬ 
pectations  in  our  contracts.  Acquisition 
decision  makers  must  shift  expectations 
as  M&S  technology  enables  more  in¬ 
formed  tradeoff  decisions  against  such 
things  as  disposal  costs.  We  must  include 
this  approach  in  our  business  case  analy¬ 
sis.  This  requires  a  fundamental  attitudi- 
nal  change  not  only  on  the  part  of  pro¬ 
gram  managers,  but  also  for  the  entire 
acquisition  team  and  DoD. 

Monitoring  needs  to  be  a  truly  inte¬ 
grated  process,  with  all  elements  actively 
involved.  A  sharing  of  analysis,  combined 
with  a  DoD  commitment  to  maximize  and 
optimize  any  potential  benefits  of  M&S 
technology,  will  bring  unprecedented 
reward — in  cheaper,  better,  stronger  prod¬ 
ucts  and  the  associated  prudence  in 
managing  the  public  purse. 

tONCIUSIONS _ 

Often,  finding  the  solution  to  complex 
technological  problems  is  a  game  of 
chance,  where  there  are  a  limited  number 
of  variables  but  a  near  limitless  combina¬ 
tion  of  these  to  bring  about  technological 
breakthroughs.  M&S  permits  the  program 
manager  to  experiment  with  a  larger  num¬ 
ber  of  possibilities  without  undue  risk. 
Once  the  essential  data  is  determined,  col¬ 
lected,  and  then  shaped  within  a  model, 


the  simulations  provide  a  tremendous 
advantage  over  traditional  methods  of  trial 
and  error. 

The  use  of  M&S  in  program  manage¬ 
ment  is  no  longer  reserved  for  programs 
on  the  cutting  edge  of  technological 
development,  nor  is  it  simply  an  “experi¬ 
mental  tool.”  Both  industry  and  the  gov¬ 
ernment  should  rely  on  sound  business 
practices  for  success.  Both  should  also 
realize  that  M&S  is  an  established  busi¬ 
ness  tool,  and  that  M&S  investment  justi¬ 
fication  should  be  based  on  a  reasoned 
cost-benefit  analysis. 

Unfortunately  establishing  the  cost- 
benefit  relation- 
shipofM&S  in¬ 
vestments  can  "The  use  of  M&S 
be  just  as  daunt-  i*1  program  moil¬ 

ing  as  the  man-  *Bem*n*  I*  no 
agement  of  a  reserved 

6  ..  ,,  for  programs  on 

program  itself  d  e 

The  use  of  a  ««  tochnologUal 

business  case  MAa. 


vestments  can  "The  use  of  M&S 

be  just  as  daunt-  In  program  mail¬ 
ing  as  the  man-  ®g*nient  is  no 

agement  of  a  reserved 

6  ..  1(  for  programs  on 

program  itself  d  e 

The  use  of  a  -  of  tmcHnological 
business  case  development,  nor 
development  |  fc  if  simply  an 

process  to  jus- 1  'experimental 
tify  M&S  de-  tool*"' 
velopment  pro¬ 
vides  a  flexible  S' 

yet  structured  methodology  for  program 
managers  to  weigh  alternatives.  Business 
case  analysis  permits  the  program  man¬ 
ager  to  justify  investment  decisions  based 
on  traditional  discounted  cash  flow  analy¬ 
ses,  as  a  function  of  externally  imposed 
constraints,  and  risk  reduction.  It  allows 
program  managers  to  capture  not  only 
those  costs  and  benefits  that  are  internally 
quantifiable  and  unquantifiable,  but  to 
address  potential  benefits  that  may  exist 
external  to  the  program.  A  disciplined  ap¬ 
proach  to  making  investment  decisions 
also  provides  a  mechanism  for  those 
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outside  the  program  management  office 
to  examine  and  assess  investments  for 
broader  applicability. 

Our  research  indicates  that  in  a  large 
section  of  the  acquisition  community, 
insufficient  rigor  is  applied  to  M&S 
investment  justification.  We  uncovered  a 
variety  of  tools  and  references  available 
to  program  managers  for  conducting  busi¬ 
ness  case  analyses.  Building  a  business 
case  not  only  helps  program  managers 
ensure  M&S  investment  is  warranted,  but 
serves  as  a  reference  for  others  when  try¬ 
ing  to  make  similar  investments  or  as  a 
baseline  document  that  other  program 
managers  and  offices  can  use  when  build¬ 
ing  investment  strategies.  Business  case 
analyses  also  build  strong  justifications  to 
defend  M&S  investments  and  bring  rigor 
and  discipline  to  the  program  or  project 
management  processes.  The  seven-step 
procedure  identified  in  the  body  of  this 
article  captures  the  essence  of  available 
guidance,  knowledge,  and  experience  and 
should  provide  program  managers  with  a 
starting  point  when  considering  M&S 
investments. 

Recommendations 


Having  a  clear  understanding  of  the 
state  of  development  we  believe  the  fol¬ 
lowing  recommendations  will  serve  to 
assist  program  managers  with  the  devel¬ 
opment  of  modeling  and  simulation 
investment  strategies  based  on  a  sound 
business  case  development  processes. 


•  Program  managers  need  to  be  encour¬ 
aged  to  add  discipline  and  structure  to 
their  M&S  justification  process.  Ser¬ 
vice  leadership  must  challenge  pro¬ 
gram  managers  to  use  business  case 
development  methodology  to  support 
M&S  investment  decisions. 

•  Program  managers  require  ready  ac¬ 
cess  to  policy  and  guidelines  from  the 
General  Services  Administration,  the 
Office  of  the  Secretary  of  Defense,  and 
each  service  in  order  to  develop  suc¬ 
cessfully  a  business  case  justification. 
We  recommend  incorporation  of  docu¬ 
ments  referenced  in  this  paper  into  the 
Defense  Acquisition  Deskbook  along 
with  a  section  to  serve  as  a  primer  for 
business  case  development. 

•  Program  managers  and  their  staff  need 
adequate  training  in  order  to  properly 
implement  business  case-based  M&S 
investment  strategy  justification.  Ac¬ 
quisition  curriculum  at  service  schools 
and  the  Defense  Systems  Management 
College  should  include  business  case 
development  familiarization  classes. 

•  Service  leadership  should  capture  suc¬ 
cess  stories  and  publish  them  in  appro¬ 
priate  service  and  DoD  journals,  maga¬ 
zines,  other  publications,  and  related 
acquisition  Internet  sites. 
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Endnotes 


1.  Quoted  by  Permission,  Brigadeer 
General  Michael  Hough,  USMC, 
deputy  program  manager  Joint  Strike 
Fighter,  during  an  address  to  the  ICAF 
Acquisition  Class,  March  19, 1999. 

2.  DoD  Total  Ownership  Cost  (TOC)  is 
the  sum  of  all  financial  resources 
necessary  to  organize,  equip,  train, 
sustain,  and  operate  military  forces 
sufficient  to  meet  national  goals  in 
compliance  with  all  laws,  all  policies 
applicable  to  DoD,  all  stnadards  in 
effect  for  readiness,  safety,  and  qual¬ 
ity  of  life,  and  all  other  official  mea¬ 
sures  of  performance  for  DoD  and  it 
components. 


3.  The  Grizzly  Program  is  a  U.S.  Army 
program.  The  Grizzly  is  a  complex 
obstacle  breaching  system  based  on 
the  MI  Abrams  tank  chassis.  It  is  de¬ 
signed  to  support  combined  arms 
maneuver  operations. 

4.  For  a  short  description  of  each,  see 
GSA’s  Information  Technology  Capi¬ 
tal  Planning  and  Investment  Guide . 
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OPINION 


THE  EVOLUTION  OF 

21st  century  acquisition 

AND  LOGISTICS  REFORM 


Paul  i.  Mcllvaine 

The  United  States  has  changed  its  military  strategy  and  stepped  up  the  use  of 
its  existing  military  forces  without  a  major  defense  budget  increase.  A  host  of 
new  initiatives  are  under  way  to  generate  the  cost  savings  necessary  to  continue 
force  modernization  without  a  major  budget  increase.  Reforms  so  far  have 
primarily  focused  on  the  acquisition  and  logistic  parts  of  the  problem.  Financial, 
contractual,  and  sustainment  reforms  are  needed  in  order  for  acquisition  and 
logistics  reform  (ALR)  to  achieve  its  full  potential. 


Joint  Vision  2010  (Shalikashvilli,  un¬ 
dated)  has  outlined  a  significant 
change  in  U.S.  military  strategy.  It 
describes  a  transition  from  a  forward- 
deployed  force  (with  stockpiles  of  mate¬ 
rials  and  permanent  troops  located  in 
anticipated  trouble  spots)  to  a  largely 
CONUS-based  force  with  a  power  projec¬ 
tion  capability  (achieved  through  rapid 
strategic  mobility  and  reduced  logistics 
tails).  Scenarios  in  which  U.S.  forces  are 
deployed  have  expanded  from  major  wars 
to  a  broad  range  of  deterrent,  conflict 
prevention,  and  peacetime  activities — in 
concert  with  our  friends  and  allies  in  al¬ 
most  all  operations.  The  Joint  Chiefs  of 
Staff  (JCS)  Focused  Logistics  Roadmap 
integrates  a  host  of  initiatives  (rapid  dis¬ 
tribution  and  response,  total  asset  visibil¬ 
ity,  information  fusion,  etc.)  designed  to 


improve  logistics  for  the  warfighter  in  sup¬ 
port  of  Joint  Vision  2010 . 

Defense  acquisition  reform  efforts  are 
under  way  to  cope  with  this  change  in 
military  strategy.  Due  to  the  high  cost  of 
supporting  existing  systems,  the  “spill¬ 
over”  of  acquisition  reform  into  the 
logistics  arena  was  a  natural  follow-on.  It 
seems  as  if  more  Department  of  Defense 
(DoD)  logistics  changes  have  been  pro¬ 
posed  in  the  past  three  years  than  in  the 
previous  30  years;  some  say  that  we’ve 
just  “scratched  the  surface.”  The  Section 
912(c)  DoD  Product  Support  Reengineer¬ 
ing  Implementation  Team,  in  its  July  1999 
Product  Support  for  the  21st  Century  re¬ 
port,  identifies  300  DoD  logistics  and 
product  support  initiatives  (Gansler, 
1999). 
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In  recognition  of  the  inseparability  be¬ 
tween  ALR  efforts,  the  fiscal  year  (FY) 
2000  National  Defense  Authorizations  Act 
changed  the  title  of  the  Under  Secretary 
of  Defense  for  Acquisition  and  Technol¬ 
ogy  to  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics 
(USD  (AT&L)). 

Current  Defense  Needs 


Current  U.S.  defense  needs  require  both 
increased  operations  and  continued  mod¬ 
ernization.  The  classical,  relatively  easy 
approach  to  achieving  the  need  has  been 
to  increase  the  defense  budget.  Contem¬ 
porary  political  pressures,  however,  have 
forced  the  search  for  more  economical 
alternatives. 

One  alternative  that  immediately  comes 
to  mind  is  to  prioritize  and  limit  opera¬ 
tions.  This  will  immediately  reduce  logis¬ 
tics  costs — the  greatest  component  of  the 
defense  budget.  Yet,  as  Representative 
George  R.  Nethercutt,  Jr.  (1996)  observed 
in  National  Defense ,  “The  Defense  De¬ 
partment  has  been  involved  in  more  de¬ 
ployments  in  the  past  7  years  than  during 
the  entire  Cold  War  ”  General  Erik 
Shinseki  (Chief  of  Staff,  Army),  in  a  Feb¬ 
ruary  statement  before  the  House  Armed 
Services  Committee,  observed  that  “Since 
1989,  the  average  frequency  of  Army  con¬ 
tingency  deployments  has  increased  from 
one  every  4  years  to  one  every  14  weeks.” 

Maintaining  near-term  readiness  of 
existing  systems  while  continuing  mod¬ 
ernization  efforts  and  reducing  infrastruc¬ 
ture  remains  a  serious  challenge.  DoD 
requests  for  additional  base  realignment 
and  closures  (BRACs)  have  yet  to  hap¬ 
pen.  Current  operating  tempos  dictate  that 


current  operations  take  precedence  over 
future  modernization.  Therefore,  it  was  no 
surprise  that  the  Joint  Aviation  Logistics 
Board  (JALB)  June  1999  report  on  Com¬ 
mercial  Support  of  Aviation  Systems  states 
that  discretionary  procurement  accounts 
dropped  by  53  percent  since  1990,  while 
operations  and  maintenance  activity  de¬ 
clined  by  only  15  percent  (Joint  Aero¬ 
nautical  Commanders’  Group,  1999).  This 
is  effectively  delaying  replacement  of 
existing  systems. 

Secretary  of  Defense  William  Cohen, 
in  the  May  1997  Report  of  the  Quadren¬ 
nial  Defense  Review ,  observed  that  “To¬ 
day,  the  Department  is  witnessing  a 
gradual  aging  of  the  force.”  This  lends 
credence  to  the  statement  in  a  1994  issue 
of  Army  RD&A  Bulletin:  “In  actuality,  our 
military  hardware  is  now  on  a  replacement 
cycle  of  about  54  years — this  in  a  world 
where  technology  typically  has  a  half-life 
from  2  to  10  years”  (Augustine,  1994). 

Figure  1  shows  that  total  operating  and 
support  costs  (for  U.S.  Mail  processing 
automation  equipment  whose  annual  sup¬ 
port  costs  equal  approximately  18  percent 
of  acquisition  cost)  are  service-life  depen¬ 
dent  and  can  approach  98  percent  of  total 
life-cycle  cost  (LCC)  if  the  equipment  is 
kept  in  service  for  54  years.  Thus,  any 
DoD  efforts  to  reduce  LCC  must  estab¬ 
lish  operations  and  support  as  a  prime 
consideration  in  designing  a  new  system 
or  improving  an  existing  system. 

The  53  percent  drop  in  procurement 
over  the  last  10  years  has  significantly 
weakened  the  Defense  Industrial  Base. 
Northrup  Grumman  chief  executive 
officer  Kent  Kresa  stated  in  March  that 
“The  trouble  with  our  industry  is,  we  have 
virtually  nothing  new.  Where  we  used  to 
have  a  new  [Pentagon  weapons  system] 
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Service  life  in  years 
PI  O&S  percent  of  LCC 

Mail  processing  automation  equipment  whose  annual  support  costs 
are  approximatelly  18  percent  of  acquisition  cost 


Figure  1.  O&S  is  Service-Life  Dependent 


coming  along,  a  major  system,  every  year 
or  every  other  year,  now  we’re  talking 
about  every  decade.  Maybe  every  two  de¬ 
cades. ..  And  if  you  happen  to  be  an  air¬ 
plane  builder,  once  the  Joint  Strike  Fighter 
gets  finished,  what’s  next?  So  how  do  you 
[attract]  the  young  person  who  wants  to 
build  airplanes,  who’ll  never  have  an  op¬ 
portunity  in  his  lifetime  to  build  an  air¬ 
plane?”  (Schneider,  2000). 

One  response  to  this  gradual  decrease 
in  modernization  is  to  exhort  managers  to 
“do  more  with  less.”  But  you  simply  can¬ 
not  do  more  of  the  same  with  less;  you 
either  do  more  with  more,  or  less  with  less. 
The  remaining  alternative  is  to  change 
procedures  and  processes  to  increase 
efficiency  and  effectiveness. 


Current  Initiatives 


Acquisition  and  logistics  reform  deals 
with  the  modernization  dilemma  by 
changing  procedures  and  processes  to  in¬ 
crease  efficiency  and  effectiveness.  Non¬ 
value-added  effort  is  eliminated.  The  goal 
is  to  free  funds  to  accomplish  needed 
modernization.  Craig  Olson,  in  a  spring 
2000  Acquisition  Review  Quarterly  arti¬ 
cle,  states  that  the  challenge  simply  can¬ 
not  be  met  short  of  a  revolutionary  change 
in  the  present  acquisition  force  structure. 
DSMC  Press  Technical  Report  TR-1-99 
states  that  it  is  too  early  to  measure  the 
success  of  acquisition  reform  (Reig, 
Gailey,  Swank,  Alfieri,  and  Suycott, 
1999).  Only  projections  can  be  made  at 
this  point. 
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The  B-52H  off  the  20th  Bomb  Squadron  "Buccaneers"  from  the  2nd 
Bomb  Wing,  Barksdale,  Louisiana. 
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Figure  2.  Defense  System  Life  Cycles 
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To  adequately  address  ALR,  we  need 
to  adopt  a  representative  time.  This  time 
perspective  is  the  entire  system  life  cycle, 
which  spans  research,  development,  test 
and  evaluation,  manufacturing  and  pro¬ 
duction,  deployment  and  materiel  field¬ 
ing,  operations  and  support,  modifications 
and  product  improvements,  and  ultimate 
disposal,  recycling,  or  demilitarization  of 
the  system. 

Figure  2  illustrates  this  time  perspec¬ 
tive  for  a  number  of  representative  defense 
systems.  The  system  life  cycle  equals  acqui¬ 
sition  time  (the  time  from  conception  of  a 
weapon  system  through  the  initial  deploy¬ 
ment  of  a  small  quantity)  plus  service  life 
(the  time  from  initial  operational  capabil¬ 
ity  to  disposal,  demilitarization,  or  recy¬ 
cling  of  the  last  system).  Jacques  Gansler, 


in  Affording  Defense,  observes  that  acqui¬ 
sition  time  varies  in  the  range  of  11  to  19 
years.  By  assuming  a  15-year  acquisition 
time  and  a  54-year  service  life,  a  repre¬ 
sentative  time  perspective  for  defense  sys¬ 
tems  can  be  defined  as  approximately  70 
years.  Some  systems,  such  as  the  R-52 
and  C-130,  have  projected  system  life 
cycles  in  excess  of  90  years. 

The  DoP  Logistics  Strategic  Plan 

In  August  1999,  Jacques  Gansler  USD 
(AT&L),  promulgated  the  year  2000  edi¬ 
tion  of  the  DoD  Logistics  Strategic  Plan 
to  modernize  our  logistics  systems  and 
improve  support  of  our  21st  century 
warfighters. 


A  US  Marine  C-130  "Hercules"  performs  a  refueling  mission  during 
an  Air  and  Amphibious  Beach  Assault  exercise* 
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Logistics  redundancy  and  duplication 
are  time-honored — albeit  inefficient — 
methods  used  in  the  past  to  support  acqui¬ 
sition  of  some  state-of-the-art  equipment 
that  has  been  largely  “long  on  perfor¬ 
mance,  but  short  on  supportability .”  Al¬ 
though  multiple  examples  of  cost-effec¬ 
tive  defense  systems  exist,  the  fact  is  that 
logistics  accounted  for  64  percent  of 
fiscal  year  1997  DoD  total  obligation 
authority  (or  its  new  synonym:  DoD  total 
ownership  cost). 

Gansler  expressed  his  dissatisfaction 
with  this  current  state  of  affairs  in  a  Janu¬ 
ary  20,  1999,  letter  (“Into  the  21st  Cen¬ 
tury:  A  strategy  for  Affordability”).  He  set 
a  goal  to  reduce  the  funding  required  by 
logistics  from  64  percent  of  total  obliga¬ 
tion  authority  in  fiscal  year  1997  to  62 


percent  by  fiscal  year  2000,  to  60  percent 
by  fiscal  year  2001,  and  a  stretch  target  of 
53  percent  by  fiscal  year  2005.  To  achieve 
this  requires  a  wholesale  recognition  that 
operations  and  support  represent  a  signifi¬ 
cant  cost  driver  which  requires  prime  con¬ 
sideration  early  and  throughout  the  design 
and  development  of  a  new  or  improved 
defense  system. 

Shortcomings  of 

Current  Measures  of  Success _ 

In  wartime,  the  cost  of  weapon  systems 
has  historically  remained  a  secondary  is¬ 
sue.  The  United  States  traditionally  pro¬ 
vides  what  it  takes  to  support  our  troops. 
During  extended  peacetime,  however,  the 


Aviation  Ordnancemen  load  an  AIM-9  "Sidewinder"  missile  on  an 
F/A-18  "Hornet"  attached  to  the  "Rampagers"  off  Strike  Fighter 
Squadron  Three  on  the  flight  deck  off  the  aircraft  carrier 
USS  George  Washington . 
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A  KC-135  follows  a  lead  vehicle  back  to  the  Alpha  ramp  at 
Grand  Forks  Air  Force  Base,  N.D*  for  a  regeneration  sortie 
and  aircrew  servicing* 


cost  of  defense  systems  tends  to  dominate 
debate.  Myopic  views  of  cost  result  in  an 
excessive  focus  on  yearly  expenditures 
(the  current  budget)  for  defense  programs. 
A  broader  strategic  perspective  involves 
the  use  of  LCC. 

The  other  measure  of  defense  systems 
success  is  effectiveness  when  used.  Despite 
the  fact  that  the  majority  of  a  defense  sys¬ 
tem’s  life  cycle  is  spent  in  peacetime,  we 
must  design  weapon  systems  for  the  worst- 
case  environment — war.  But  war  is  the 
world’s  most  uneconomical  undertaking. 

If  we  combine  the  two  measures  above, 
the  “bottom  line”  of  all  improvement 
efforts  can  be  summed  up  as  cost-effec¬ 
tiveness.  But  cost-effectiveness  is  a  judg¬ 
ment  call — a  subjective  versus  objective 


measure.  This  ensures  that  continued  con¬ 
troversy  will  remain  an  integral  part  of  the 
defense  acquisition  process  now  and  in  the 
future. 

The  Worst  and  Best  Practices  - 
A  Comparison 


The  best  illustration  of  new  ALR  ef¬ 
forts  is  to  compare  the  “worst  practices” 
of  the  1960s  through  the  1980s  with  the 
“best  practices”  (including  ALR  initia¬ 
tives)  of  the  1990s  through  2010  and  be¬ 
yond.  Figure  3  illustrates  this  comparison 
over  a  70-year  lifespan,  including  each 
“cradle-to-grave”  year  drawn  to  scale. 
This  is  the  time  needed  to  demonstrate 
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21**  Century  Acquisition  and  Logistics  Reform  for  a  typical  Program 
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Figure  3  (continued). 

21**  Century  Acquisition  and  Logistics  Reform  for  a  typical  Program 
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how  well  today’s  ALR  efforts  achieve  their 
objectives. 

The  ordinate  represents  yearly  expen¬ 
ditures  on  a  constant  dollar  basis.  Thus, 
Figure  3  as  a  hypothesis  only  addresses 
the  economic  success  of  ALR.  By  follow¬ 
ing  the  money  from  cradle-to-grave,  we 
can  illustrate  the  economic  success  of  a 
“reformed”  program,  in  which  identical  sys¬ 
tem  effectiveness  is  assumed.  Consequent¬ 
ly,  the  measure  of  the  economic  success 
of  ALR  is  minimizing  the  area  under  the 
curve  (LCC — or  its  new  synonym, 
defense  systems  total  ownership  cost). 

The  Worst  Practices  of 
THE  1960s  THROUGH  THE  1980S 


The  engineer  in  the  1960s  through 
1980s  could  be  dubbed  “The  Lone 
Ranger.”  This  engineer  practiced  the  art 
of  sequential  engineering:  First,  design  a 
system  to  work  and  meet  or  exceed  all 
point  design  requirements.  LCC — a  minor 
concern — was  dutifully  reported  as  the 

result  of  de¬ 
sign.  Subse¬ 
quently,  design 
engineers  ad¬ 
dressed  manu¬ 
facturing  and 
producibility. 
If  “show-stop- 
ping  mistakes”  were  identified  (such  as 
designing  a  wing  2  inches  longer  than  the 
largest  fixture  in  the  tool  inventory),  then 
a  redesign  or  engineering  change  was 
made,  requiring  more  money  and  time. 

Logistics  considerations  then  followed, 
demonstrating  the  attitude  that  the  “log- 
gies”  would  simply  support  whatever  the 
design  engineers  created.  Their  measure 


"The  engineer  in 
the  1960s  through 
1980s  could  be 
dubbed  'The  Lone 
Ranger/" 


of  worth  was  dealing  with  shortsighted  de¬ 
sign  decisions,  and  keeping  systems  (no 
matter  how  good  or  bad)  running  in  the 
field.  If  show-stopping  mistakes  were 
identified  (such  as  inaccessible  parts  re¬ 
quiring  periodic  replacement),  then  a  re¬ 
design  or  engineering  change  was  made, 
again  requiring  more  money  and  time. 

Following  this  path,  the  next  major  item 
was  deployment — the  system  was  turned 
over  to  the  user.  The  user’s  last  involve¬ 
ment  with  the  program  was  in  the  “front 
end”  requirements  determination.  User 
point-design  requirements  were  consid¬ 
ered  inviolate  and  further  discussions  with 
users,  or  “challenges”  of  questionable  user 
requirements  during  development,  were 
frowned  upon.  If  the  user  identified  a 
show-stopping  mistake  upon  receiving  the 
deployed  system  (such  as  a  nonwaterproof 
system  targeted  for  exposed  storage),  then 
an  engineering  change  was  incorporated, 
again  requiring  more  money  and  time. 
Alternately,  if  the  change  was  not  made, 
then  the  tactical  and  operational  logisti¬ 
cians  were  sent  scurrying  to  find  un¬ 
planned,  unbudgeted,  covered-storage 
space  for  the  system.  And  eventually,  after 
a  number  of  changes  or  other  accommo¬ 
dations  (collectively  called  MOD  1  or  the 
A  Model),  the  user  got  what  the  user 
wanted — sort  of. 

The  steps  in  sequential  engineering 
could  be  dubbed  re-engineering  or,  better 
yet,  “getting  it  right  the  second,  third  and 
fourth  times.”  Too  often,  the  downstream 
result  was  lower  operational  availability 
[readiness]  at  higher  cost. 

This  paper-intensive,  iterative  process 
resulted  in  “islands  of  automation”  for 
each  functional  discipline,  a  high  cost  to 
change,  and  slower  implementation.  A 
general  defense  “rule  of  thumb”  was  that 
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An  M-l  13  ambulance  off  the  1st  Armored  Division  on  a  trail 
near  site  "Gator/'  Bosnia-Herzegovina. 


technical  manuals  averaged  $1,000  per 
paper  page  to  change  or  update.  During 
visits  to  operating  bases,  I  personally  ob¬ 
served  maintenance  technicians  with 
handwritten  notebooks  in  which  they 
documented  their  “tricks  of  the  trade  .” 
These  notebooks  helped  them  to  better 
deal  with  problems  that  were  inadequately 
described  in  the  official  technical  manuals 
or  ignored  by  the  designers. 

After  deployment,  the  program  man¬ 
ager  executed  a  “program  management 
responsibility  transfer”  to  another  orga¬ 
nization,  usually  identified  as  a  “logistics 
command”  or  equivalent.  And  the  origi¬ 
nal  program  management  team  returned 
to  new  system  development  to  again  re¬ 
peat  the  process.  The  receiving  command 
then  performed  the  function  of  program 
management — often  mislabeled  as  “logis¬ 
tics  management” — for  the  remainder  of 
the  system  life  cycle. 


This  mislabeling  was  due  to  the  one 
certainty  in  acquisition — constant  change. 
Program  management  of  a  brand  new  sys¬ 
tem  includes  the  functions  of  design,  test, 
production,  fielding,  operations  and  sup¬ 
port,  and  eventual  disposal.  Logistics 
management  primarily  centers  on  the 
function  of  support  or  sustainment.  After 
deployment,  a  defense  system  will  cer¬ 
tainly  face  change.  Modifications  include 
the  functions  of  design,  test,  production, 
fielding  and  installation,  operations  and 
support,  and  eventual  disposal — virtually 
identical  functions  of  program  manage¬ 
ment.  Thus,  the  art  and  science  of  program 
management  is  the  one  function  consis¬ 
tently  performed  throughout  the  entire 
system  life  cycle  for  both  new  starts  and 
modifications  to  existing  systems. 

Systems  produced  via  sequential  engi¬ 
neering  usually  needed  long,  big,  logis¬ 
tics  tails  to  compensate  for  design  short- 
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comings.  An  organic  mindset  usually  per¬ 
meated  all  logistics  thinking  and  planning. 
The  politically  correct  language  to  de¬ 
scribe  contractor  support  was  “interim 
contractor  support” — even  if  it  lasted  for 
decades.  Higher  parts  inventories  were 
pre-positioned  within  “arms  length”  to 
compensate  for  long  logistics  cycle  and 
repair  times  (with  great  distance  between 
places  of  use  and  repair).  Organic  deliv¬ 
ery  had  to  wait 
its  turn  as  a 
“competing 
priority  ”  Lim¬ 
ited  asset  vis¬ 
ibility  con¬ 
fused  and  ob¬ 
fuscated  logis¬ 
tics  efficien¬ 
cies  by  people 
not  knowing  where  a  part  was  in  the  pipe¬ 
line  or  warehouse,  or  its  condition.  Sup¬ 
ply  chain  management  under  this  scenario 
is  and  was  haphazard  and  inefficient.  The 
net  result  was  usually  lower  operational 
availability  (synonym:  readiness)  at  a 
greater  cost. 

But  the  popular  method  to  deal  with 
low  operational  availability  and  high  costs 
was  to  change  the  system.  The  quest  to 
redesign  is  aptly  named.  The  goal  was  to 
make  the  system  do  what  it  should  have 
done  in  the  first  place.  Unplanned  prod¬ 
uct  improvements  and  engineering  change 
proposals  were  legion — especially  as  new 
technology  allowed  both  performance  im¬ 
provement  and  support  improvement. 

As  time  marched  on,  however,  the  sys¬ 
tem  eventually  entered  the  wearout 
phase — the  end  of  its  service  life.  Many 
sequentially  engineered  systems  were  de¬ 
signed  for  service  lives  of  15  or  20  years, 
at  which  point  the  replacement  systems 


"Systems  produced 
via  sequential 
engineering  usually 
needed  long,  big, 
logistics  tails  to 
compensate  for 
design  short¬ 
comings/' 


were  planned  to  arrive  and  allow  smooth 
transition  from  old  to  new  systems  before 
the  added  costs  of  wearout.  This  planning 
optimism  could  result  in  disappointment 
if  replacement  systems  were  still  mired 
in  the  early  acquisition  phases — years 
from  deployment.  In  this  case,  the  only 
practicable  alternative  was  to  institute  an 
unplanned  service  life  extension  program 
(SLEP).  This  could  prove  quite  costly  if 
the  system  was  not  originally  designed  for 
rebuild — as  in  many  cases. 

But  given  enough  time  and  money, 
“anything  could  be  accomplished.”  Be¬ 
sides,  the  political  visibility  of  modifica¬ 
tion  programs  has  traditionally  been  less 
than  new  starts.  Thus,  the  A  model,  B 
model,  etc.,  became  the  modernization 
norm  as  technology  continued  its  inevi¬ 
table  march  forward.  With  increased  ser¬ 
vice  lives  becoming  commonplace,  at  least 
one  or  two  unplanned  and  expensive 
SLEPs  could  be  expected  for  many  typi¬ 
cal  legacy  systems.  Gansler,  in  Affording 
Defense,  speaks  of  the  “...tradition  of 
keeping  equipment  in  the  field  approxi¬ 
mately  twice  as  long  as  it  was  initially 
planned...”  (1989). 

After  production  termination  (postpro¬ 
duction  support),  one  of  the  major  down¬ 
sides  of  new  technological  improvement 
became  evident.  Sole  source  suppliers  of 
unique  spare  parts  for  legacy  defense  sys¬ 
tems  that  were  no  longer  in  production 
faced  higher  costs  to  produce  fewer  quan¬ 
tities  when  orders  fell.  Thus,  suppliers 
were  faced  with  decisions  to  either  raise 
costs  and  keep  their  spare  parts  produc¬ 
tion  lines  open,  or  embrace  new  technol¬ 
ogy  and  cease  production  of  the  older 
technology.  With  little  or  no  post  produc¬ 
tion  support  planning — evidenced  by  sole 
sources  of  supply,  high  costs,  slow 
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response  times,  and  the  like — the  govern¬ 
ment  was  often  faced  with  reverse  engi¬ 
neering  of  replacement  parts,  life  of  type 
buys,  unplanned  depot  overhauls,  or  sys¬ 
tem  redesign.  Advantages  gained  by  im¬ 
proved  technology  (even  with  logistics 
benefits)  could  be  more  than  offset  by  the 
difficulty  of  changing  systems  that  were 
not  designed  for  easy  change. 

Eventually,  all  systems  reach  the  end 
of  their  economic  or  physical  service  lives, 
when  replacing  the  system  represents  the 
most  cost-effective  alternative.  Disposal 
of  the  legacy  system  is  then  addressed — 
usually  for  the  first  time.  Museum  dona¬ 
tions,  foreign  military  sales,  retirement  to 
Davis-Monthan  Air  Force  Base,  etc.,  rep¬ 
resented  relatively  easy  solutions  to  the 
problem. 

Hazardous  materials  used  in  the  origi¬ 
nal  design  of  the  system  presented  spe¬ 
cial  problems.  If  additional  monies  could 
be  found  for  this  unplanned  activity, 
proper  disposal  of  hazardous  materials 
was  feasible.  Shortsighted  solutions  to  the 
problem  involved  burial  in  remote  parts 
of  military  bases,  burial  at  sea,  or  other 
environmentally  unfriendly  alternatives 
that  are  no  longer  acceptable.  However, 
this  previous  practice  allowed  a  quick, 
cheap,  and  easy  end  to  the  system  as  the 
program  was  terminated.  Decades  later, 
the  legacy  of  shortsighted  hazardous  ma¬ 
terials  disposal  became  evident.  But  with 
no  program  around,  it  was  likely  that 
somebody  else  would  have  to  bear  the  cost 
and  danger  of  cleanup. 

This  rather  pessimistic  view  indeed  rep¬ 
resents  a  compilation  of  “worst  practices” 
for  a  theoretical  program.  In  the  early 
1970s,  my  engineering  supervisor  once 
told  me  to  “get  real”  and  stop  creating 
controversy  and  delay  by  trying  to  design 


it  right  the  first  time.  He  said  the  only  way 
to  build  a  defense  system  was  to  spend 
the  money  quickly,  or  some  other  program 
would  “steal”  it.  If  the  design  wasn’t  quite 
right  the  first  time,  you  would  eventually 
get  the  money  and  commitment  to  fix  it 
later! 

Unfortunately,  the  wisdom  in  his  com¬ 
ments  reflected  the  inflexibility  of  DoD’s 


financial  sys-  r 

terns  and  the  in- 
.  ,  4 

centives  and  re¬ 
wards  in  place 
at  the  time.  De-  • 
spite  these  short- 1 
sighted  incen-  * 
fives,  many  pro 
grams  in  the 
1960s  through 


"Eventually^  all 
systems  reach 


the  end  ef  their 
ecenentic  or  physical 
service  lives,  when 
replacing  the  system 
represents  the 
most  cost-effective 
alternative/' 


1980s  used  dif¬ 
ferent  methods  and  better  practices.  These 
are  the  defense  systems  that  are  still  in 
service  today.  Longevity  is  (and  remains) 
the  bellwether  of  a  good  system  design. 


The  Best  Practices  of 

THE  1990s  THROUGH  THE  201  OS 


Choices  characterize  the  “front  end”  of 
modem  acquisition  practice.  Computer- 
aided  design,  manufacturing  and  logistics 
tools  allow  faster  computation  of  design 
tradeoffs,  assessment  of  design  alterna¬ 
tives,  and  completion  of  design — com¬ 
pared  with  previous  manual  practices.  But 
these  modern  tools  can  be  used  in  two 
ways: 

Complete  the  Process  Faster 
and  at  Lower  Front-end  Cost 

Alternative  1  yields  the  most  immedi¬ 
ate  and  measurable  results  and  is  the 
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objective  in  stated  goals  to  reduce  cycle 
times.  But  the  DSMC  Press  Technical 
Report  TR-1-99  states  that  the  pre-1989 
average  duration  of  the  engineering  and 
manufacturing  development  phase  was  6.5 
years,  rising  to  8.7  years  during  the  period 
1993-1996,  and  dropping  to  8.3  years  in 
1997  (Rieg  et  al.,  1999).  This  data  could 
lead  to  a  conclusion  of  limited  progress 
to  date. 

Complete  the  Process  at  the 
Same  Speed  and  Cost,  but  Better 

Alternative  2  is  to  complete  the  process 
in  the  same  amount  of  time  (by  pursuing 
additional  design  iterations  to  yield  a 
higher  quality  and  reduced  risk  design) 


and  at  the  same  front-end  cost  (by  rein¬ 
vesting  any  front-end  savings  in  high  pay¬ 
off  areas  for  downstream  cost  reduction). 

Alternative  2  has  the  potential  to  yield 
even  greater  LCC  savings.  This  alterna¬ 
tive  makes  comparisons  much  clearer,  and 
it  is  depicted  in  Figure  3.  If  Alternative  1 
were  depicted,  the  gray  line  in  the  chart 
would  move  to  the  left  (reduced  cycle 
time)  and  move  down  (reduced  front-end 
cost).  Reduced  oversight  would  have  the 
same  effect. 

Since  the  one  certainty  in  all  of  acqui¬ 
sition  is  constant  change  and  inevitable 
technological  progress,  open  systems 
design  has  emerged  as  one  of  the  most 
intelligent  design  innovations  in  decades. 


UH-1  Huey  helicopters  from  the  37th  Helicopter  Flight  prepare 
to  land  in  a  field  during  a  simulated  Launch  Facility  (LF) 
Recapture  Exercise. 
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Open  systems  design  recognizes  that 
change  is  inevitable  and  seeks  to  estab¬ 
lish  system  architectures  and  system  de¬ 
signs  with  the  flexibility  and  partitioning 
to  facilitate  future  change  to  the  maximum 
extent  practicable.  Where  needed  tech¬ 
nologies  have  not  yet  matured,  preplanned 
product  improvements  and  technology 
insertions  can  be  more  easily  implemented 
downstream  with  open  systems  design. 
This  technique  should  prove  to  be  of  great 
benefit  to  systems  design,  production,  and 
logistics  support. 

The  engineer  in  the  1990s  through 
2010s  could  be  dubbed  “design  team 
leader”  or  “systems  engineer.”  This  mod¬ 
em  engineer  practices  the  art  of  concur¬ 
rent  engineering  (systematic  consideration 
of  all  elements  of  the  system  life  cycle — 
including  manufacturing  and  support — 
from  the  beginning).  Integrated  product 
and  process  development  (IPPD)  is  the 
predominant  technique  that  simulta¬ 
neously  integrates  all  disciplines  through 
the  use  of  multidisciplinary  teams  in  each 
area  that  ultimately  has  a  hand  in  the  ac¬ 
quisition  and  design  process.  This  in¬ 
cludes  software  engineers,  production 
engineers,  logistics  engineers,  test  engi¬ 
neers,  reliability  engineers,  contract  spe¬ 
cialists,  financial  managers,  LCC  analysts, 
user  representatives,  business  managers, 
contractor  personnel,  and  others. 

The  goal  is  a  producible,  supportable, 
cost-effective  design  that  satisfies  user  re¬ 
quirements  at  the  lowest  practicable  LCC. 
The  objective  is  to  “get  it  right  the  first 
time”  through  a  series  of  design  iter¬ 
ations — each  of  which  yields  a  higher 
quality  design.  What  we  call  show-stop¬ 
ping  mistakes  should  be  rare,  since  all 
participants  in  the  process  are  team 
members  from  the  outset. 


r‘iy •:  .  ' 


Inflexible  point  design  requirements 
have  been  replaced  with  thresholds  (mini¬ 
mum  acceptable  values)  and  objectives  (a 
more  desirable 
value  to  work 

toward).  This  ;  "The  engineer  in 
range  of  accept- 1  the  1990s  through 

able  values  al- !  2010*  could  be 
lows  enhanced  f  *«»>bed  'design 

design  flexibil-  »•»«  !•■«*«' « 
ity  in  mating:  "S'— 

reasoned  trade¬ 
offs.  Greater  user  involvement  in  all 
phases  of  the  acquisition  program  allows 
continual  dialogue  and  “challenge”  of  user 
needs  to  foster  more  intelligent  design 
decisions. 

Cost  as  an  independent  variable  (CATV) 
methodologies  are  used  to  acquire  afford¬ 
able  DoD  systems  by  a  better  balance 
among  performance,  schedule,  and  LCC. 
Tradeoff  analysis  is  repeatedly  applied 
throughout  the  system  life  cycle  as  the  key 
tool  that  results  in  LCC  reduction — not 
just  reporting. 

The  tendency  of  keeping  defense  sys¬ 
tems  in  service  at  least  twice  s  long  as  ini¬ 
tially  planned  can  be  dealt  with  by  more 
realistically  designing  systems  for  longer 
service  lives  or  designing  systems  for 
periodic  rebuild  (if  more  cost-effective). 
Costly,  unplanned  service  life  extension 
programs  can  be  radically  reduced.  Open 
systems  design  is  expected  to  make  modi¬ 
fications,  technology  insertions,  and  prod¬ 
uct  improvements  less  expensive  and 
faster  to  implement. 

The  measure  of  the  worth  of  logisti¬ 
cians  is  no  longer  how  well  they  deal  with 
easily  avoidable  downstream  problems 
(reactive  problem  solving).  Proactive 
problem  prevention  is  the  primary  goal 
and  focus  of  the  acquisition  logistician. 
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Heavy  Expanded  Mobility  Tactical  Truck  (HEMTT) 


Problem  solving  for  the  tactical-oper¬ 
ational  logistician  will  never  entirely 
disappear,  but  will  be  reduced  to  manage¬ 
able  proportions,  due  to  greater  logistics 
attention  in  the  front  end  of  the  process. 

Producibility  and  supportability  Engi¬ 
neering  Change  Proposals  (ECPs)  are 
drastically  reduced  as  a  result  of  the  con¬ 
tinuous  involvement  of  produciblity 
engineers  and  acquisition  logisticians 
from  the  outset  of  the  program. 

The  entire  acquisition  process  will  be 
tied  together  via  an  integrated  data  envi¬ 
ronment  that  allows  all  users  real-time  ac¬ 
cess  to  the  data  they  need.  Enhanced  com¬ 
munication  among  program  participants 
facilitates  improved  management.  Lower 
cost  to  change  documents  and  faster 
implementation  (a.k.a.  reduced  cycle 
time)  should  become  the  norm,  because 
of  this  paperless  process. 


As  a  result  of  continuous  user  involve¬ 
ment  in  the  program,  deployment  can  be 
expected  to  be  much  smoother.  The  user 
knows  what  to  expect  from  the  system  and 
the  developer  knows  how  the  user  will  op¬ 
erate  and  support  this  system. 

The  Section  912(c)  report  on  Program 
Manager  Oversight  of  Life  Cycle  Support 
identified  30  pilot  programs  to  test  the 
concept  of  the  program  manager  assum¬ 
ing  life-cycle  system  responsibility 
(PMOLCS  Study  Group,  1999).  Presum¬ 
ing  success,  the  program  manager  of  the 
future  can  be  expected  to  retain  the  life- 
cycle  responsibility  for  the  system  subse¬ 
quent  to  deployment.  Field  commanders 
and  users  will  most  likely  continue  their 
responsibility  for  the  present — i.e.,  the 
readiness  and  sustainability  of  the  fielded 
system.  The  program  manager  can  be 
expected  to  assume  responsibility  for  the 
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future — i.e.,  improvement  of  the  system 
(to  include  the  operational  subsystem  and 
the  logistics  support  subsystem). 

Systems  designed  and  produced  via 
concurrent  engineering  or  IPPD  usually 
need  shorter  logistics  tails  to  accompany 
high  quality  designs.  Partnering  arrange¬ 
ments  with  industry  for  long-term  life- 
cycle  contractor  support — governed  by 
sensible  “win-win”  business  relation¬ 
ships — will  replace  the  “organic  mindset.” 
Necessary  organic  support  is  still  retained 
for  those  functions  defined  as  core. 

Total  asset  visibility  (TAV)  provides 
users  with  timely  and  accurate  informa¬ 
tion  on  the  location,  movement,  status,  and 
identity  of  units,  personnel,  equipment, 
and  supplies.  This  tremendously  increases 
logistics  efficiency  and  supply  chain  man¬ 
agement  because  all  players  in  the  pro¬ 
cess  know  where  each  part  is  (pipeline, 
warehouse,  contractor  plant,  etc.)  and  in 
what  condition  each  part  is  (ready  for  in¬ 
stallation,  undergoing  repair,  etc.).  Rapid 
transportation  and  distribution  (coupled 
with  prime  or  direct  vendor  delivery)  re¬ 
sults  in  lower  total  parts  inventories  and 
improved  logistics  responsiveness.  The 
net  result  will  be  higher  operational  avail¬ 
ability  (readiness)  at  lower  cost. 

Mutually  acceptable  measures  of  effec¬ 
tiveness  (MOEs)  in  a  performance-based 
environment  are  the  basis  upon  which 
sound  business  relationships  and  part¬ 
nering  agreements  are  built.  Defense  con¬ 
tractual  arrangements  in  the  future  will  de- 
emphasize  units  repaired  per  month,  in¬ 
ventory  quantities,  etc.  New  defense  con¬ 
tractual  arrangements  will  evolve  to  per¬ 
formance-based  logistics  in  which  we  buy 
operational  availability  or  customer  wait 
time  and  mean  logistics  delay  time. 


Costs  per  operating  hour  is  another 
MOE,  common  in  commercial  industry, 
that  can  be  expected  to  see  increased 
Defense  use  in  the  future.  Incentivization 
of  improvements  to  systems  and  sub¬ 
systems  will  be  the  norm  in  which  con¬ 
tractors  can  reap  additional  profit  by 
delivering  added  amounts  of  “goodness” 
in  systems. 

A  single  maintenance  data  collection 
system  among  all  the  services  must  evolve 
in  order  for  ALR  to  reach  its  full  promise. 
Accurate  and  timely  field  data  is  the 
cornerstone  upon  which  sustaining  engi¬ 
neering  efforts  assess  current  fielded 
system  effectiveness  and  address  priori¬ 
tized  improvements  to  both  the  system  and 
its  support. 

As  systems  age,  multiple  sources  of 
supply  are  still  available  because  fewer 
single-source,  single-application  sub¬ 
systems  and  components  were  used  in  the 
basic  system  design.  Modernization 
through  spares  (MTS)  is  a  strategy  of  tech¬ 
nology  insertion  and  the  use  of  commer¬ 
cial  products,  processes,  and  practices  to 
extend  a  com-  J 

ponent’s  useful :  "System*  designed 
life  or  improve  a„d  produced  via 
its  reliability,  concurrent  engi- 
MTS  can  ex- :  neering  or  IPPD 
tend  a  defense  usually  need  shorter 
system’s  useful  logistic*  tails  to 
life  by  dealing  accompany  Wgh 
with  the  prob-  floolity  designs." 

lem  of  newer  • 

technology  supplanting  the  older  technol¬ 
ogy  in  legacy  defense  systems.  Good, 
solid  post-production  support  planning 
will  not  preclude  all  modernization  and 
supply  problems,  but  instead  will  reduce 
the  remaining  problems  to  manageable 
proportions. 
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As  the  system  enters  the  wearout  phase, 
disposal  plans  are  put  into  action  and  dispo¬ 
sal  budgets  are  executed.  Fewer  hazardous 
materials  should  be  used  in  the  original 
design.  Disposal  planning  for  those  that 
were  used  can  be  updated  before  the 
system  is  ultimately  disposed. 

This  rather  optimistic  view  represents 
“best  practice.”  In  about  70  years  (the  life 
cycle  for  the  average  defense  system),  we 
will  be  able  to  accurately  measure  the 
results  of  today’s  ALR  efforts. 


Summary 


Exhortations  for  government  to  “adopt 
commercial  methods”  and  “do  it  like  in¬ 
dustry  does  it”  are  common.  The  Section 
912[c]  report  on  A  Plan  to  Accelerate  the 
Transition  to  Performance-Based  Services 
reiterates  the  future  projection  that  “Gov¬ 
ernment  acquisition  spending  will  con¬ 
tinue  to  be  used  to  foster  socioeconomic 

goals”  (Ander¬ 
son,  1999).  But 
the  report  goes 
on  to  observe 
that  “Because 
the  goals  of  a 
DoD  source 
selection  differ 
from  those  in  a 
commercial 
source  selec¬ 
tion,  the  most 
aggressive  commercial  methods  are  not 
appropriate  for  DoD  use.”  Thus,  industry 
has  different  goals,  fewer  constraints,  and 
more  flexible  financial  systems  that  are, 
in  many  ways,  less  burdensome  than  those 
used  by  government.  Industry  does  not 
always  do  it  faster,  cheaper  and  better  than 


"Industry  does  not 
always  do  it  faster, 
cheaper  and  better 
than  government-— 
a  more  accurate 
statement  might  be 
that  industry  does 
it  differently  than 
government/' 


government — a  more  accurate  statement 
might  be  that  industry  does  it  differently 
than  government. 

“Pay  me  now  or  pay  me  later”  remain 
the  chief  options  in  DoD  acquisitions.  Past 
defense  acquisition  systems  tended  to 
incentivize  the  “pay  me  later”  option — 
evidenced  by  the  high  cost  of  operating 
and  maintaining  our  existing  systems. 
Defense  ALR  efforts  are  simply  asking  for 
a  more  reasoned  tradeoff  among  the  “pay 
me  now  or  pay  me  later”  alternatives  for 
future  systems,  while  we  struggle  with  the 
legacy  of  our  past  decisions. 

Best  commercial  practice  involves  giv¬ 
ing  good  program  managers  a  clear  job 
with  wide  latitude,  minimal  oversight,  and 
considerable  flexibility  in  making  invest¬ 
ment  decisions  in  their  commercial  prod¬ 
uct  programs,  while  holding  them  account¬ 
able  for  results.  Investing  “up  front”  mon¬ 
ies  to  avoid  “downstream”  expenditures 
must  always  be  the  subject  of  a  serious 
business  case  analysis  resolved  through 
tradeoff  decisions  involving  a  long-term 
perspective.  Government  program  man¬ 
agers  for  defense  programs  need  that  same 
latitude,  flexibility,  and  accountability. 

Acquisition  and  Logistics 
Reform  (ALR)  Recommendations 

Acquisition  and  logistics  reform  efforts 
to  date  have  made  considerable  progress 
and  hold  great  promise.  However,  much 
work  remains  to  be  done  in  order  to 
achieve  the  full  potential  of  these  reforms. 
Below  are  six  areas  where  improvements 
can  be  made. 

First,  government  contracting  tools 
need  to  change  to  reflect  the  new  reality. 
DoD  Directive  5000.1  and  DoD  Regula¬ 
tion  5000.2-R  direct  that  long-term,  life- 
cycle  contractor  support  is  the  preferred 
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method  of  logistics  support.  The  JALB 
Report  Commercial  Support  of  Aviation 
Systems  cites  the  framework  of  the  World 
Airlines  and  Suppliers  Guide :  The  maxi¬ 
mum  parts  cost  guarantee  is  an  agreement 
whereby  suppliers  provide  a  parts  cost 
(Joint  Aeronautical  Commanders’  Group, 
1999). 

To  adopt  this  standard  commercial 
practice  in  government  may  be  viewed  as 
limiting  competition.  Long-term,  life- 
cycle  contractor  support  requires  innova¬ 
tive  multiyear  service  contract  arrange¬ 
ments,  possible  statutory  changes,  and 
logistics  contractual  strategies  that  en¬ 
compass  longer  defense  service  lives  (54 
years,  for  example).  In  current  practice, 
government  contracts  are  of  much  shorter 
duration  and  can  hamper  government 
program  managers  from  more  efficiently 
executing  life-cycle  responsibility. 

Second,  a  long-term  financial  perspec¬ 
tive  (approaching  54  years)  is  necessary 
to  implement  ALR.  The  planning,  pro¬ 
gramming,  and  budgeting  system  (PPBS) 
is  the  Defense  Department’s  financial  sys¬ 
tem  that  provides  the  “fuel”  to  make  ALR 
work.  But  the  PPBS  looks  forward  about 
six  years  at  most.  Paul  Mann,  in  Aviation 
Week  and  Space  Technology  (2000),  ob¬ 
serves  that  “. .  .unsynchronized  Pentagon/ 
congressional  budget  cycles,  result  in  ar¬ 
tificial  cost  projections  and  an  acquisition 
culture  of  intellectual  dishonesty.”  Thus, 
government  financial  reform  has  not  kept 
pace  with  ALR  efforts. 

Third,  government  program  managers 
who  can  obtain  a  great  return  on  invest¬ 
ment  of  “up  front”  RDT&E  monies  to  sig¬ 
nificantly  reduce  downstream  operations 
and  maintenance  (O&M)  monies  are  still 
thwarted  in  their  attempts  to  make  seri¬ 
ous  tradeoff  decisions.  “Colors  of  money” 


and  the  intractability  of  the  current  PPBS 
may  defeat  a  compelling  government  busi¬ 
ness  case  analysis  for  up-front  investment 
to  greatly  reduce  downstream  expendi¬ 
tures.  A  commercial  producer  would 
readily  adopt  this  same  business  case 
analysis.  Procedures  that  allow  program 
managers  to  retain  and  reinvest  savings 
(or  portions  thereof)  in  their  programs  are 
needed.  In  today’s  environment,  govern¬ 
ment  program  managers  are  still 
incentivized  to  minimize  expenditure  of 
scarce  RDT&E  and  procurement  funds; 
i.e.,  their  current  budgets — not  optimize 
LCC. 

Fourth,  commercial  practices  and  use 
of  commercial  or  nondevelopmental 
items,  if  prop¬ 


"'Colors  of  money' 
and  the  intracta¬ 
bility  of  the  current 
PPBS  may  defeat  a 
compelling  govern¬ 
ment  business  case 
analysis  for  up¬ 
front  investment 
to  greatly  reduce 
downstream 
expenditures*" 


erly  applied,  are 
great  methods 
of  improving 
processes  and 
reducing  costs. 

Commercial 
practices  are 
different  from 
defense  prac¬ 
tices,  however, | 
and  usually  in-  f 
volve  shorter 
life  cycles  with 

little  customer  control.  So  customers  must 
have  the  flexibility  to  react  to  the  market 
decisions  of  their  commercial  suppliers. 
For  example,  when  a  commercial  supplier 
decides  that  new  technology  no  longer 
makes  it  cost-effective  to  support  a  legacy 
system,  customers  are  forced  to  either 
set  up  their  own  logistics  support  or 
replace  their  older  systems  with  the  latest 
technology. 

But  the  government  financial  system 
dictates  the  use  of  different  colors  of 
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"It  takes  time 
to  spend  money 
wisely;  but,  in  the 
current  environ- 
ment,  slower, 
wiser  spending  is 
impracticable." 


money  for  each  alternative.  If  the  govern¬ 
ment  program  manager  expected  the  tech¬ 
nology  cycle  to  be  longer  and  budgeted 
operations  and  maintenance  monies  to 
support  a  legacy  system,  these  budgeted 

monies  (even  if 
adequate  to  re¬ 
place  the  older 
system  with 
the  latest  tech¬ 
nology)  cannot 
be  used  to  fi¬ 
nance  procure¬ 
ment  of  the  lat¬ 
est  technology 
system.  They  can  only  be  used  to  operate 
and  maintain  the  older  system.  This  cur¬ 
rent  state  of  affairs  will  often  force  gov¬ 
ernment  program  managers  into  “short¬ 
sighted”  and  uneconomical  decisions  dic¬ 
tated  by  an  inflexible  PPBS,  rather  than 
by  logic. 

Fifth,  “haste  makes  waste.”  When  con¬ 
stant  defense  budget  turbulence  threatens 
to  decrease  or  eliminate  needed  program 
funds,  Industry  reacts  by  hesitating  to 
make  any  long-term  capital  investment  in 
the  program.  Program  managers  react  to 
this  financial  instability  by  hurrying  to 
spend  monies  before  another  budget  cut. 
It  takes  time  to  spend  money  wisely;  but, 
in  the  current  environment,  slower,  wiser 
spending  is  impracticable.  An  old  friend 
and  colleague  of  mine  (currently  serving 
as  a  government  program  manager)  re¬ 
cently  commented  that  the  most  common 
question  he  is  asked  at  reviews  is  “What  are 
your  obligation  and  expenditure  rates?” 

This  budget  turbulence  is  not  limited 
to  DoD.  A  front  page  Washington  Post 
article  on  a  lost  Mars  polar  lander  stated 
that  “ . . .  NASA’s  efforts  to  tighten  the  bud¬ 
get  screws  and  encourage  certain  kinds  of 


risk  taking — under  a  philosophy  known 
as  ‘faster,  cheaper,  better’ — finally  went 
too  far”  (Sawyer,  2000).  “...Managers 
may  have  failed  to  raise  alarms  more 
clearly  up  the  chain  of  command  because 
of  concern  that  they  would  lose  ground  in 
the  competition  for  tight  funding. . 

Sixth,  past  inattention  to  the  sustain¬ 
ment  and  maintenance  (operations  and 
support)  phase  of  the  system  life  cycle  has 
clouded  the  objectives  of  this  phase:  readi¬ 
ness  and  sustainment  of  the  system,  and 
improvement  of  the  system  (to  include  the 
operational  subsystem  and  the  logistics 
support  subsystem). 

In-service  engineering  analysis  of 
fielded  systems  performance  is  hampered 
by  “islands  of  automation”  (that  inhibit 
the  free  flow  of  information)  and  unique 
maintenance  data  collection  systems  that 
have  evolved  for  each  service.  Greater 
program  management,  engineering,  and 
logistics  resources  and  attention  devoted 
to  this  area  will  result  in  decreased  LCC. 
Improved  maintenance  data  accuracy,  bet¬ 
ter  data  collection  and  dissemination  effi¬ 
ciency,  enlightened  data  reduction  and 
analysis,  prioritization  of  engineering 
improvements  with  both  operational  and 
logistics  benefits,  and  better  visibility  into 
product  support  costs  and  logistics  effec¬ 
tiveness  are  prerequisites  to  executing 
modern  program  manager  life-cycle  re¬ 
sponsibility  and  achieving  the  objectives 
of  this  phase. 


Conclusion 


Much  effort  has  been  expended  on 
ALR.  Government  financial  systems, 
however,  have  not  been  reformed,  and 
currently  support  the  old  way  and  worst 
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practices  in  doing  business.  Thus,  ALR  ef¬ 
forts  will  fail  at  worst  and  stumble  at  best 
under  the  statutes,  policies,  processes,  pre¬ 
cedents,  and  procedures  of  the  current  fi¬ 
nancial  systems  applicable  to  defense  in 
the  legislative  and  executive  branches  of 
government.  Financial  reform,  contractual 
innovation,  and  other  changes  to  provide 


defense  program  managers  greater  flexi¬ 
bility  in  the  expenditure  of  funds  and 
greater  incentivization  to  minimize  LCC 
is  sorely  needed  to  complete  the  transfor¬ 
mation  of  the  DoD  acquisition  and  lo¬ 
gistics  system  from  the  20th  to  the  21st 
century. 
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TUTORIAL 


COST  AS  AN 

INDEPENDENT  VARIABLE: 
PRINCIPLES  AND 
IMPLEMENTATION 


Col  Mithael  A.  Kaye,  USAF,  Lt  Col  Mark  5.  Sobota,  USAF, 
David  R.  Graham,  and  Allen  L  Gotwald 

Cost  as  an  independent  variable  is  a  key  tool  in  the  thrust  to  reduce  total 
ownership  cost  for  defense  systems.  While  the  need  for  CAIV  is  driven  by  cost 
constraints,  success  relies  upon  identification  and  use  of  viable  performance, 
cost,  schedule,  and  risk  “trade  space.”  The  Air  Force  has  integrated  CAIV 
concepts  with  those  in  the  Reduction  in  Total  Ownership  Program  (R-TOC), 
and  has  published  a  comprehensive  guidebook  for  better  understanding. 


The  Defense  System  Affordability 
Council  (DSAC)  Strategic  Plan  es¬ 
tablished  Goal  2  to  lower  the  total 
ownership  cost  (TOC)  of  defense  prod¬ 
ucts.  The  plan  further  established  separate, 
aggressive  objectives  under  that  goal  for 
systems  in  acquisition  and  fielded  sys¬ 
tems.  These  goals  are  further  emphasized 
in  the  draft  new  DoD  5000.1  and  2. 

To  provide  a  focal  point  on  all  reduc¬ 
tion  in  TOC  (R-TOC)  efforts,  encompass¬ 
ing  weapon  system,  infrastructure,  and 
indirect  dimensions,  the  Air  Force  estab¬ 
lished  an  R-TOC  program  office  (SAF/ 
AQXT).  S  AF/AQXT  and  the  authors  colla¬ 
borated  to  publish  the  R-TOC  Guidebook, 


which  integrated  Cost  as  an  Independent 
Variable  (CAIV)  and  a  comprehensive  R- 
TOC  process  for  fielded  systems  (1999). 
The  R-TOC  process  relies  on  baselining 
operating  and  support  costs,  identifying 
TOC  drivers,  and  identifying  R-TOC 
opportunities.  CAIV  drives  system  design 
decisions  by  providing  comprehensive 
information  on  alternatives  and  impacts. 

Whereas  CAIV  and  the  R-TOC  process 
have  many  principles  in  common,  CAIV 
exerts  the  most  leverage  when  it  influences 
system  design  and  the  R-TOC  process  is 
most  effective  on  fielded  systems.  The 
relationship  is  shown  in  Figure  1 . 
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Figure  1.  R-TOC/CAIV  Effectiveness 


CAIV  Construct 


CAIV  is  a  key  strategy  for  implement¬ 
ing  R-TOC  in  the  acquisition  process,  and 
is  particularly  effective  during  system 
development.  Air  Force  Instruction  (AFI) 
10-601  (1998)  defines  CAIV  as  “the  pro¬ 
cess  of  using  better  business  practices, 
allowing  trade  space  for  industry  to  meet 
user  requirements,  and  considering  opera¬ 
tions  and  maintenance  costs  early  in 
requirements  definition  in  order  to  procure 
systems  smarter  and  more  efficiently.” 

CAIV  is  founded  upon  two  primary 
principles:  First,  system  costs  are  con¬ 
strained.  Whereas  some  programs  do  ob¬ 
tain  additional  funding  when  needed,  such 
funding  is  often  at  the  expense  of  other 


programs  or  future  modernization.  Sec¬ 
ond,  “trade  space”  is  the  foundation  for 
smart  decisions.  Trade  space  is  the  range 
of  alternatives  available  to  decision  mak¬ 
ers.  It  is  four-dimensional,  comprising 
performance,  cost  (TOC),  schedule,  and 
risk  impacts. 

The  Air  Force  established  a  set  of  tenets 
that  are  core  to  CAIV  implementation. 
The  concept  of  well-understood  trade 
space  is  the  capstone  tenet  that  enables 
decisions  critical  to  meeting  user  needs 
while  reducing  TOC.  The  remaining  five 
tenets  are  the  pillars  that  enable  trade 
space  to  be  defined  and  exploited.  Figure 
2,  the  CAIV  model,  depicts  the  relation¬ 
ship  of  the  CAIV  tenets. 
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Figure  2.  CAIV  Model 


Trade  Space 


CAIV  provides  better  support  for  criti¬ 
cal  decisions  by  identifying  viable  perfor¬ 
mance,  schedule,  cost  (TOC),  and  risk 
trade  space.  Identification  and  use  of 
viable  trade  space,  or  the  range  of  alter¬ 
natives,  with  full  knowledge  of  real  and 
potential  impacts,  is  essential  for  making 
the  right  decisions  to  meet  user  needs 
while  reducing  TOC.  CAIV  employs  a 
hierarchy  of  cost  reduction  opportunities 
and  tradeoffs  to  meet  aggressive  cost  tar¬ 
gets,  first  looking  to  improve  acquisition 
and  sustainment  efficiencies,  then  scruti¬ 
nizing  noncritical  requirements.  Tradeoffs 
of  critical  performance  requirements  are 


only  to  be  addressed  as  a  last  resort,  with 
the  agreement  of  the  Milestone  Decision 
Authority  and  user. 

Trade  space  is  commonly  defined  by 
alternatives  in  terms  of  the  performance, 
cost,  and  schedule  impacts  that  each 
alternative  presents.  Risk  must  also  be 
included  in  two  ways.  First,  risk  is  a  fourth 
dimension  in  the  trade  space,  recognizing 
that  critical  decisions  may  be  driven  by 
the  risks  of  certain  alternatives.  Second, 
risk  actually  “discounts”  the  anticipated 
performance,  cost,  and  schedule  options; 
in  other  words,  it  lessens  the  trade  space 
to  ensure  a  decision  maker  does  not  trade 
away  something  that  may  not  be  attain¬ 
able.  For  example,  assume  you  have  a 
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system  with  anticipated  range  of  2000 
miles  versus  a  requirements  threshold  of 
1500  miles.  You  could  trade  away  up  to 
500  miles  of  range  for  a  fully  tested,  vali¬ 
dated  system  and  still  meet  threshold. 
However,  you  definitely  would  not  trade 
away  500  miles  of  range  at  the  beginning 
of  program  definition  and  risk  reduction 
(PDRR),  when  there  are  potential  weight 
growths,  fuel  consumption  increases,  etc. 

Figure  3  portrays  the  cost-performance 
trade  space  for  a  key  performance  param¬ 
eter  (KPP),  characterized  by  threshold  and 
objective  values.  Note  that  Figure  3  shows 
a  “risk  reserve”  line  to  depict  the  amount 
that  the  trade  space  is  restricted,  to  pre¬ 
vent  trading  away  what  is  not  yet  realized. 
The  “solution  set”  line  represents  the 


optimum  cost-performance  combinations: 
Points  in  the  shaded  region  are  solutions, 
but  for  any  given  point,  either  more  per¬ 
formance  for  the  same  cost  or  the  same 
performance  for  less  cost  is  possible. 

The  trade  space  is  of  course  multidi¬ 
mensional,  corresponding  to  the  number 
of  KPPs.  Tradeoffs  can  be  performed  at 
many  levels.  In  the  example  above,  where 
a  KPP  is  involved,  the  user  must  agree  to 
the  tradeoff.  When  a  contractor  has  con¬ 
figuration  control  below  the  “A  Spec,” 
then  the  contractor  can  make  tradeoff 
decisions  as  long  as  the  A  Spec  is  met. 
The  key  is  that  the  decision  maker  must 
fully  understand  impacts  on  the  other  ele¬ 
ments,  especially  cost  (TOC),  in  the  trade 
space. 


Figure  3.  Cost-Performance  Trade  Space 
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Capabilities-Basep  Requirements 

CAIV  relies  upon  capability-based  re¬ 
quirements.  Implementation  requires  the 
user  to  define  capability-based  require¬ 
ments,  stating  what  the  system  needs  to 
do  instead  of  how  to  build  the  system  and 
how  subsystem  allocations  are  made.  Such 
definition  allows  the  system  development 
team  flexibility  to  define  a  best-value  sys¬ 
tem  meeting  user  requirements.  This  re¬ 
quires  the  development  of  operationally- 
oriented  performance  requirements  with 
a  minimum  number  of  KPPs.  Require- 
ments-setting  authorities  must  take  special 
efforts  to  exclude  requirements  not  direct¬ 
ly  contributing  to  user  needs.  Prioritizing 
requirements  helps  exclude  nonessential 
requirements  while  helping  system  devel¬ 
opers  maximize  use  of  the  trade  space  by 
focusing  on  characteristics  contributing 
most  to  mission  accomplishment. 

The  requirements  process  specifies  that 
system  requirements  be  reviewed  at  each 
milestone  and  revised  as  needed.  But  sys¬ 
tem  performance  thresholds  defined  at 
Milestone  I  (MS  I),  prior  to  Phase  I  trade 
studies,  may  be  difficult  to  change  in  later 
reviews.  That  is  the  most  important  rea¬ 
son  for  introducing  the  cost  dimension 
into  the  requirements-setting  process  as 
early  as  possible.  Beginning  with  realis¬ 
tic  and  feasible  levels  of  requirements, 
within  the  best  available  measures  of  cost 
estimation,  provides  the  ability  to  identi¬ 
fy  a  realistic  performance  requirements 
baseline.  Through  CAIV  trades,  the  pro¬ 
gram  can  take  advantage  of  alternative 
approaches  and  designs  to  achieve  higher 
levels  of  performance  at  the  same  or  low¬ 
er  levels  of  cost  as  more  information  al¬ 
lows  cost  estimates  to  become  more 
refined  and  accurate.  Staying  flexible  in 


the  finalization  of  requirements  is  impor¬ 
tant,  as  emphasized  in  the  draft  new  DoD 
5000.1  and  (2000). 

The  operational  requirements  docu¬ 
ment  (ORD)  I  should  identify  system 
characteristics  and  define  threshold  ranges 
required  for  user  effectiveness  and  be 
treated  as  interim  versus  final.  Warfighters 
and  users  use  mission  effectiveness  and 
cost  performance  analyses  as  key  parts  of 
the  analysis  of  alternatives  (AoA).  When 
the  preferred  ap¬ 
proach  is  iden¬ 
tified,  users  em-  "Warfighters  and 
ploy  mission  ef-  users  use  mission 
fectiveness  ana- !  effectiveness  and 
lysis  and  CAIV  |  cosf  performance 


principles  to  set 


analyses  as  key 


.  ...  ,  ,  parts  of  the  analysis 

imtial  weapon  J, 

system  require- 
ments,  based  on  If 
the  best  insight 

available  to  TOC.  As  a  result,  ORD  I 
should  include  TOC  objectives. 

Phase  I  trade  studies  should  provide  re- 
quirements-setters  sufficient  insight  to 
TOC/LCC  impacts  for  them  to  set  specific 
threshold  levels  that  ensure  both  mission 
effectiveness  and  affordability.  During 
Phase  I,  the  weapon  system  IPT  uses 
CAIV  to  define  TOC  impacts  and  conduct 
trade  to  refine  requirements  studies  fo¬ 
cused  on  design  and  sustainment.  The 
derived  data  enables  users.  The  IPT  en¬ 
sures  a  continued  ability  to  meet  baseline 
requirements  while  adapting  to  require¬ 
ments  evolutions  that  drive  system  modi¬ 
fications.  Finalization  of  requirements 
from  the  ranges  defined  in  ORD  I  should 
occur  at  this  time.  Both  the  Joint  Strike 
Fighter  (JFS)  and  Advanced  Amphibious 
Assault  Vehicle  (AAAV)  have  employed 
this  evolutionary  approach. 
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Figure  4.  Partnering 


Partnering 


We  can  no  longer  allow  the  sequential, 
isolated  approach  to  developing  systems! 
CAIV  relies  on  partnered  management  of 
the  trade  space  between  the  user,  acquirer, 
and  industry  participants,  including  strong 
involvement  from  the  sustainment  experts 
in  each  of  the  three  communities. 


In  the  past,  the  all-too-common  process 
flow  was  sequential,  with  limited  part¬ 
nering.  Under  CAIV,  the  user  defines  sys¬ 
tem  requirements,  with  comprehensive 
input  from  sustainers.  In  addition,  the 
acquirer  and  industry  partners  support  the 
user  by  identifying  and  quantifying  the 
major  risks  and  TOC  drivers,  thereby 
enabling  better  informed  decisions.  The 
acquirer  leads  system  development,  with 
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strong  user  and  sustainer  involvement  on 
potential  trades  identified  by  the  acquirer 
or  industry.  Industry,  under  contractual 
incentives,  allocates  requirements  and 
designs  systems  for  minimized  TOC, 
seeks  lower  cost  and  equally  capable 
atematives  for  system  elements  under  its 
control,  and  makes  recommendations  for 
elements  not  under  its  control  (based  on 
trade  studies). 

A  key  element  in  CAIV  success  will 
be  the  cost  performance  integrated  prod¬ 
uct  team  (CPIPT).  The  CPIPT  will  com¬ 
plete  cost-performance-schedule  tradeoffs 
leading  to  CATV-based  cost,  performance, 
and  schedule  objectives.  The  CPIPT,  to 
include  the  user,  must  work  closely  to¬ 
gether  to  agree  on  final  threshold  and  ob¬ 
jective  values  for  cost,  performance,  and 
schedule  parameters  in  preparation  for  the 
milestone  decision  and  completion  of  key 
documentation. 

CAIV-based  threshold  and  objectives 
for  KPPs,  life-cycle  cost  targets,  and  criti¬ 
cal  milestones  become  the  core  drivers  for 
the  subsequent  program.  These  core  pa¬ 
rameters  must  be  consistent  across  the 
ORD,  acquisition  program  baseline 
(APB),  acquisition  strategy,  and  test  and 
evaluation  master  plan  (TEMP)  that  come 
together  beginning  with  MS  I.  The  IPT  and 
CPIPT  remain  active  through-out  the  pro¬ 
gram  life  cycle  up  through  Phase  III,  pro¬ 
duction,  fielding  and  deployment,  and 
operational  support.  The  CPIPT  is  the 
cognizant  group  to  continue  CAIV-based 
cost-performance-schedule  trade-offs,  to 
establish  cost  range  objectives  for  pro¬ 
duction  and  sustainment,  and  to  revise 
performance,  cost,  and  schedule  objectives 
prior  to  each  milestone  decision. 

After  MS  I,  the  contractor  should  be 
a  major  contributor  for  CAIV-based 


analyses  and  tradeoffs.  Solicitations  need 
to  address  life-cycle  cost  and  performance 
objectives  and  request  industry’s  approach 
for  implementing  and  managing  the  CATV 
process.  The  contractor  and  government 
managers  should  give  consideration  to 
establishing  a  co-chaired  hierarchy  IPT 
structure.  Working  level  IPTs,  formed 
around  critical  subsystems,  report  to  a  man¬ 
agement  EPT  through  a  systems  integration 
IPT. 

The  core  of  the  CAIV  process  is  trade¬ 
offs  conducted  by  the  working  level  IPTs. 
Working  level  IPTs  should  be  given  a  cost 
target  and  chartered  to  conduct  cost-per¬ 
formance  tradeoffs  to  reduce  subsystem 
cost  drivers.  Cost-performance  tradeoffs 
at  the  system  level  depend  directly  on  sub- 
system  level  cost  and  performance 
tradeoffs.  The  : 
number  and  fo¬ 
cus  of  IPTs  will  "CAIV  relies  on 
change  as  the  ;  acceptance  of 
program  ma- 1  higher  risk  to 
tures  from  de-  *99™ssh,ely 
velopment  into  «  "best 

production  and  for  |||e  user»  ,* 
operation  and 
sustainment. 

CAIV  relies  on  acceptance  of  higher 
risk  to  aggressively  pursue  a  “best  value” 
system  for  the  user.  Contractors  and  IPTs 
should  be  given  incentives  to  conduct  ef¬ 
fective  and  meaningful  cost  performance 
tradeoffs. 

The  contractor  is  key  in  cost-perfor¬ 
mance  trades.  The  contractor-government 
partnership,  in  which  the  customer  em¬ 
ploys  prudent  risk  management  and  par¬ 
ticipates  fully  in  the  development  of  the 
confidence  needed  to  entrust  weapon  sys¬ 
tem  development,  ensures  the  success  of 
the  contractor’s  cost-performance  trades. 


"best 
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This  partnership  is  the  natural  evolution 
of  the  insight-oversight  paradigm. 

Promotions  and  assignment  policies 
must  recognize  and  reward  successes  and 
best  efforts.  Cost-type  contracts  should  in¬ 
clude  an  award  fee  clause  that  shares  cost 
savings  between  the  government  and  con¬ 
tractor.  Incentives  were  a  key  ingredient 
in  the  Peace  Shield  program’s  total  suc¬ 
cess  in  meeting  aggressive  schedule  and 
performance  requirements.1 

The  contractor  shared  40  percent  of  a 
$50  million  incentive  bonus  with  employ¬ 
ees.  The  incentive,  combined  with  an 
integrated  earned  value-metrics-schedule 
system,  superb  partnering  with  the  acqui¬ 
sition  office  and  customer,  and  aggressive 
software  management,  overcame  an  early 
schedule  deficit  to  deliver  a  validated 
system  six  months  early. 

There  are  many  tools  available  to  mo¬ 
tivate  contractors:  competitions  between 
primes,  competition  at  the  component 

breakout  level, 
award  fees, 
award  terms, 
performance 
bonuses,  value 
engineering 
opportunities, 
and  multiyear 
or  sole-source 
awards.  Acquisition  offices  should  address 
incentives  as  part  of  the  CAIV  plan  in¬ 
cluded  in  the  acquisition  strategy.  Contrac¬ 
tors  must  be  encouraged  to  meet  and 
exceed  life-cycle  cost  reduction  targets, 
not  just  near-term  cost  objectives. 

In  order  to  address  life-cycle  costs  up 
front  and  early,  for  example,  continuation 
of  a  multiyear  contract  can  be  tied  to 
reaching  or  exceeding  interim  life-cycle 
cost  targets.  The  contractor  could  receive 


"Promotions  and 
assignment  policies 
must  recognize  and 
reward  successes 
and  best  efforts*" 


a  larger  percentage  of  the  next  production 
lot  or  could  be  awarded  the  next  phase 
if  production  unit  costs  meet  or  exceed  a 
cost  reduction  profile.  The  objective 
should  be  to  tie  life-cycle  cost  targets  and 
contractor  performance  together  through 
innovative  and  aggressive  incentives. 


TOC/LCC  FOCUS 


CAIV  requires  all  team  members  main¬ 
tain  focus  on  TOC/LCC.  Fiscal  constraint 
is  a  reality  that  all  Air  Force  stakeholders 
must  recognize.  Based  on  the  determina¬ 
tion  of  resource  availability,  stakeholders 
must  set  an  aggressive  but  realistic  TOC 
target  for  the  system. 

At  each  milestone,  decision  makers  will 
review  targets  and  progress  toward  veri¬ 
fying  that  they  will  be  met.  Cost  targets 
shall  be  addressed  in  the  acquisition  strat¬ 
egy  and  will  be  included  and  tracked  in 
the  acquisition  program  baseline  (APB). 
CAIV-based  cost  targets  should  be  in¬ 
cluded  in  requests  for  proposals  (RFPs) 
and  contractors  given  incentives  to 
achieve  cost  targets.  Also,  all  personnel 
must  constantly  be  cognizant  of  the  need 
to  identify  cost  reduction  opportunities 
and  tradeoffs. 

Typical  targets  for  procurement  and 
sustainment  are  average  unit  procurement 
cost  (AUPC  equals  total  procurement 
funding/total  quantity)  and  average  unit 
O&S  cost  (AUO&SC  equals  unit  cost  per 
flight  hour,  etc.).  Each  of  these  metrics 
can  be  tailored  to  the  specific  system.  Pro¬ 
curement  targets  can  be  expressed  as  a  cost 
profile;  for  example,  AUPC  versus  pro¬ 
duction  lot  number.  Sustainment  targets  may 
be  expressed  as  a  percentage  reduction 
relative  to  O&S  costs  of  a  similar  system. 
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Confidence  limits  for  development, 
procurement,  and  sustainment  cost  targets 
will  vary  with  the  phase  of  the  program 
as  well  as  system  complexity  and  the  de¬ 
gree  to  which  it  is  pushing  the  state  of  the 
art.  At  MS  I,  for  example,  development 
cost  estimates  should  be  more  accurate 
than  estimates  for  O&S  costs. 

Estimating  TOC/LCC  poses  a  signifi¬ 
cant  challenge  early  in  the  program,  es¬ 
pecially  for  cutting-edge  programs.  In 
addition,  government  and  industry  have 
sometimes  had  large  variances  in  cost 
estimates,  leading  to  program  start  com¬ 
plications.  SAF/AQ  has  sponsored  a  cost 
estimating  reinvention  team  to  address 
some  of  these  challenges. 

Unfortunately,  TOC/LCC  focus  may 
well  be  the  weakest  link  in  the  CAIV 
process.  The  reality  is  that  instant-year 
dollars  are  politically  and  practically  the 
driving  factor  in  program  decisions.  It  is 
easy  to  say  decision  makers  must  make 
decisions  based  on  TOC  considerations, 
but  a  lot  harder  to  manage  real  programs 
that  way.  When  we  see  high  payoff  invest¬ 
ments  rejected  in  our  programs,  it  seems 
we  often  manage  by  the  “modified 
Wimpie  philosophy,”  which  is  not,  “I  will 
gladly  pay  you  on  Tuesday  for  a  ham¬ 
burger  today,”  but  “I  will  gladly  pay  you 
for  three  hamburgers  on  Tuesday  for  a 
hamburger  today!”  While  we  may  bemoan 
such  an  approach,  we  must  realize  that 
without  the  hamburger  today,  we  may 
starve  and  not  reach  Tuesday!  The  only 
way  we  can  overcome  this  malady  is  with 
more  accurate,  believable  tradeoff  data,  so 
that  decision  makers  will  have  a  better 
picture  of  the  true  impact. 


Risk-Based  Management 


Risk  management  is  an  integral  part  of 
CAIV.  It  recognizes  we  cannot  afford  to 
avoid  all  risk,  but  rather  must  manage 
the  critical  risks.  A  comprehensive  and 
disciplined  risk  management  program 
throughout  a  program’s  life  cycle  is  criti¬ 
cal  to  effective  management  to  meet  cost, 
performance,  and  schedule. 

As  established  by  DoD  5000.2-R  (DoD, 
1998)  and  AFMCP  63-101,  the  risk  man¬ 
agement  program  identifies  and  tracks  risk 
drivers,  generates  risk-handling  plans,  and 
provides  for  monitoring  to  track  risk  “re¬ 
tirement”  or  growth.  Risk  reduction  mea¬ 
sures  are  included  in  cost-performance 
tradeoffs,  where  applicable. 

Program  partners  must  jointly  identify, 
analyze,  and  prioritize  critical  program 
risks,  then  periodically  review  handling 
plan  progress.  ^  ;  ^ 

Handling  ap- 

proaches  can  "The  reality  is 
run  the  gamut  that  instant-year 

from  develop-  dollars  are 
mg  alternate  de- 1  politically  and 
signs  for  critical  practically  the 
components  to  driving  factor  In 
simple  monitor-  Pr*9r«m  decisions." 

ing  to  ensure  a 
risk  does  not 

take  root  and  grow.  A  commonly  used  tool 
for  identifying  and  prioritizing  risks  is 
shown  in  Figure  5. 

The  risk  matrix  helps  identify  the  risks 
that  must  be  addressed — those  that  have 
high  probability  of  occurrence  and  poten¬ 
tial  high  impact.  Risk  management  is  not 
just  an  engineering  function!  The  team 
must  address  programmatic  risks  and  all 
functions  must  be  involved  in  the  handling 
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Figure  5.  Risk  Matrix 


plans  as  appropriate.  In  fact,  any  program 
element  associated  with  cost,  schedule, 
and  performance  has  a  direct  interface 
with  the  risk  management  process. 

It  is  important  to  remember  that  risk 
management  is  used  throughout  a  pro¬ 
gram’s  life  cycle.  A  risk  management  plan 
should  be  part  of  the  early  CAIV  strat¬ 
egy.  Several  programs  have  reaped  ben¬ 
efits  by  conducting  government-industry 
risk  brainstorming  and  prioritization  ses¬ 
sions  that  directly  influenced  the  RFP 
(e.g.,  Space  Lift  Range  Modernization  and 
Satellite  Control  Network  programs).  RFPs 
should  require  offerors  to  identify  their 
risk  management  approach,  risks  inherent 
in  their  design,  and  risk  handling  plans. 

Measurement 


Setting  realistic  but  aggressive  goals  for 
cost  and  performance  is  a  key  element  of 
program  management.  Measuring  progress 


toward  attaining  those  goals  is  a  challeng¬ 
ing  but  critical  element  of  CAIV  imple¬ 
mentation.  Proper  metrics  will  add  value 
by  aiding  decision  making  rather  than  sim¬ 
ply  reporting  status.  Metrics  measure  a 
wide  range  of  parameters,  including  health 
of  critical  processes,  effectiveness  of  cost 
saving  initiatives,  and  status  of  the  “value 
stream.” 

The  earned  value  management  system 
(EVMS)  and  technical  performance  mea¬ 
surements  (TPMs)  provide  particularly 
useful  metrics.  A  contractor  needs  an 
EVMS  to  manage  complex  tasks.  Proper 
partnering  will  provide  government  of¬ 
fices  insight  into  the  EVMS.  TPMs  are 
product  design  and  performance  assess¬ 
ments  that  estimate  values  of  essential 
performance  parameters  of  the  current 
design.  TPMs  assist  in  determining  the  po¬ 
tential  impacts  of  differences  between 
planned  and  actual  values.  Table  1  lists 
other  useful  metrics  and  tools. 
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Table  1.  Illustrative  CAIV  Metrics  and  Features 


Are  cost  objectives  defined 
and  consistent  with  require¬ 
ments  programmed  and 
projected  fiscal  resources? 

•  Out-year  resources  identified?  (dollars) 

•  Production  and  O&S  cost  objectives  included  in  the 

RFP? 

•  Key  tradeoff  issues  addressed  (e.g.,  in  AoA)? 

Is  the  government  managing 
to  achieve  cost  objectives? 

•  RFP  contains  a  strict  minimum  number  of  performance 
specifications?  (number) 

•  CPIPT  functioning,  tradeoff  space  identified  in  program 
baseline  and  RFP? 

•  Risks  to  achieve  cost  objectives  identified  and  program 
steps  to  address  these  defined?  (risk  plan  ) 

•  Incentives  for  achieving  life  cycle  cost  objectives 
included  in  the  RFP  and  contract?  (  percent  relative 
total  contract  dollars;  period  of  performance  tied  to  life 
cycle  cost  target  profile) 

•  Mechanism  for  contractor  suggestions  to  reduce 
production  and  O&S  costs  in  place  and  operating? 

(value  engineering  clause) 

•  Allocation  of  cost  objectives  provided  to  IPTs  and  key 
suppliers 

•  Measurement  and  estimation  of  reliability  and  maintain¬ 
ability 

•  Robust  contractor  incentives  plan  in  place? 

Are  contractors  managing  to 
achieve  cost  objectives? 

•  Provide  appropriate  tools  for  cost-performance  tradeoffs 
(including  incentives)  and  participate  in  cost-perfor¬ 
mance  tradeoff  process  (hierarchy  IPT  structure;  award 
fee  flow  down  to  IPT  members) 

•  Identifies  and  implements  new  technologies  and  manu¬ 
facturing  processes  that  can  reduce  costs 

•  Identifies  procedural/process  impediments  to  cost 
reduction  measures 

•  Establishes  strong  relationship  with  vendor  base, 
including  sound  incentives  structure 
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These  examples  reflect  the  degree  to 
which  a  program  is  structured  for  CAIV 
success.  They  provide  important  and  ob¬ 
servable  tools  that  assist  in  setting  aggres¬ 
sive  production  and  operating  and  support 
(O&S)  cost  objectives  and  management 
of  the  program.  In  some  cases,  quantita¬ 
tive  metrics  can  be  defined  as  indicated 
by  the  parentheses  at  the  end  of  a  process 
step. 


CAIV  Evolution  Through 
the  Life  Cycle 


CAIV  is  initiated  at  the  beginning  of  a 
program  and  evolves  through  the  life 
cycle.  As  an  acquisition  methodology, 
the  application  of  CAIV  varies  between 
each  acquisition  phase  as  performance 
requirements,  initially  established  as  a 
range  between  required  thresholds  and 
desired  objectives,  narrow  as  a  program 
migrates  toward  production.  Let’s  briefly 
examine  the  application  of  CAIV  across 
an  acquisition  life  cycle  using  the  CAIV 
tenets. 


Phase  0 

In  Phase  0,  CAIV  supports  the  analy¬ 
sis  of  alternatives  (AOA)  by  focusing  on 
KPPs  and  cost  drivers.  The  R-TOC  pro¬ 
cess  provides  essential  tools  with  data 

from  existing 
systems  and 
techniques  for 
baselining  cost 
drivers.  At  the 
end  of  this 
phase  the  ORD 
is  established 
with  threshold  and  objective  ranges,  fund¬ 
ing  is  assigned  for  development  (where 


"CAIV  is  initiated 
at  the  beginning 
off  a  program  and 
evolves  through 
the  life  cycle." 


necessary),  and  a  maintenance  strategy  is 
proposed  to  minimize  long-term  O&S. 
Unfortunately,  current  practice  provides 
a  detailed  ORD,  usually  based  on  mission 
effectiveness  analysis,  without  adequate 
assessment  of  TOC  impacts. 

Requirements.  The  highest  Air  Force 
levels  ensure  that  defense  system  effec¬ 
tiveness  is  optimized  within  constraints  of 
available  and  projected  resources.  Also, 
warfighters  and  users  utilize  mission 
effectiveness  and  cost  performance  analy¬ 
ses  as  key  parts  of  the  AoA.  When  the 
preferred  approach  is  identified,  users  em¬ 
ploy  mission  effectiveness  analysis  and 
CAIV  principles  to  set  initial  weapon 
system  requirements,  based  on  the  best 
insight  available  to  total  ownership  cost 
(TOC).  As  a  result,  ORD  I  should  include 
TOC  objectives. 

Partnering.  The  warfighter  and  user 
lead  the  requirements  definition  efforts, 
and  should  have  acquirer  support  if  an 
existing  program  office  or  a  development 
planning  core  team  is  assigned. 

TOC  focus.  The  Phase  0  IPT  focuses 
on  the  life-cycle  aspects  of  a  program  by 
developing  and  recommending  an  opera¬ 
tion  and  sustainment  (O&S)  strategy  that 
will  optimize  the  elements  of  reliability 
and  maintainability.  TOC  focus  recog¬ 
nizes  that  the  majority  of  life-cycle  cost 
is  in  the  O&S  phase,  and  the  resulting 
strategy  will  reflect  a  system  that  both 
maximizes  system  reliability  and  requires 
minimum  downtime  to  fault,  isolate  and 
repair  problems. 

Risk-based  management.  As  much 
acquirer  involvement  as  possible  is  de¬ 
sirable  to  support  the  user  in  gaining  an 
early  understanding  of  potential  risks  and 
handling  alternatives.  In  preparation  for 
Milestone  1,  the  acquisition  strategy  must 
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identify  the  key  risk  areas  within  the 
proposed  solution.  These  will  become 
integrated  into  the  CAIV  strategy  and, 
ultimately,  the  RFP. 

Measurement.  Objective  TOC  should 
be  set  in  the  ORD.  The  requirements 
“owner”  should  estimate  which  require¬ 
ments  are  likely  to  be  cost  drivers;  and 
for  each  such  driver,  track  whether  ad¬ 
equate  TOC  insight  has  been  gained.  This 
is  hard!  For  that  reason  it  is  essential  that 
users  work  with  financial  experts  within 
acquiring  agencies  to  best  estimate  total 
ownership  costs. 

Phase  I 

Maximum  leverage  for  CAIV  efforts 
exists  in  Phase  I,  since  system  design  has 
not  yet  been  finalized. 

Requirements.  During  Phase  I,  the  wea¬ 
pon  system  IPT  uses  CAIV  to  define  TOC 
impacts  and  conduct  trade  studies  focused 
on  design,  particularly  as  influenced  by 
O&S.  Tradeoff  data  enables  warfighters 
and  users  to  refine  requirements. 

Partnering.  The  team  now  typically  in¬ 
cludes  an  acquisition  program  office  and 
at  least  one  contractor.  The  acquirer  will 
have  the  lead  for  completing  PDRR,  but 
must  ensure  close  coordination  with  the 
user.  Using  CAIV,  the  team  should  base¬ 
line  expectations  through  the  following 
steps: 

•  Identify  common  and  unilateral  ob¬ 
jectives  to  ensure  concerted,  focused 
effort. 

•  Define  all  interdependencies  and  orga¬ 
nize,  plan,  and  commit  to  act  to  meet 
them.  Central  to  this  aspect  is  analysis 
of  the  contractor’s  network  and  criti¬ 
cal  path,  and  integrated  master  plan 


(IMP).  Based  on  this  analysis,  the 
government  should  build  its  own  IMP 
to  ensure  government  action  meets 
timelines  expected  by  the  contractor  and 
does  not  give  any  cause  for  contractor 
claims. 

•  Generate  a  unified  risk  handling  plan. 

•  Develop  a  concept  of  operations 
(CONOPS)  to  define  management  and 
working  level  interaction,  metrics,  etc. 

TOC  focus.  With  a  funded  and  manned 
program,  adequate  effort  can  now  be  ex¬ 
pended  to  identify  TOC  and  risk  impacts 
from  design  and  sustainment  trade  stud¬ 
ies. 

Risk-based  management.  The  team 
should  generate  a  unified  risk-handling 
plan  to  address 
high  and  mod- 

erate  risk  areas.  |,e  team  should 
The  plan  should  generate  a  unified 
consolidate  pre-  risk-handling  plan 

vious  risk  ana- §  to  address  high 
lyses,  conduct  and  moderate 
further  brain- 1  risk  areas/' 
storming,  pri-  gVjgg 
oritize  risks,  al¬ 
locate  risk-handling  responsibilities,  and 
regularly  review  status.  Implementation 
may  include  provisions  in  acquisition 
strategies,  contracts,  or  parallel  efforts.  As 
a  minimum,  the  contract  should  require 
the  contractor  to  conduct  trade  studies  to 
identify  trade  space. 

Measurement.  The  team  must  con¬ 
tinue  to  track  TOC  estimates  to  TOC  ob¬ 
jectives.  In  addition,  requirements  “own¬ 
ers”  should  continue  to  track  degree  of 
insight  to  TOC  impact  for  each  identified 
cost  driver.  The  team  should  track  risk 
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drawdown  per  the  handling  plans  with 
metrics  tied  to  specific  risk  reduction 
accomplishments.  Use  of  TPMs  can  be 
very  effective  for  such  effort. 

Phase  II 

Early  in  Phase  II,  system  design  freeze 
limits  the  ability  of  CATV  to  generate  fur¬ 
ther  substantial  cost  savings.  However, 
focus  on  producibility  and  O&S  may  still 
yield  benefits. 

Requirements.  The  IPT  uses  CAIV  to 
ensure  requirements  are  met  at  minimum 
TOC. 

Partnering.  The  team  may  expand  to  in¬ 
clude  other  organizations  (e.g.,  AFOTEC). 
The  same  principles  of  teaming  apply  as 
applied  in  Phase  I. 

TOC  focus.  Tradeoff  studies  will  focus 
on  manufacturing  and  sustainment  im¬ 
pacts  on  TOC  and  risk.  A  mature  design 
and  testing  of  components  and  prototypes 
should  enable  more  accurate  estimates  of 
O&S  impacts  on  TOC. 

Risk-based  management.  The  con¬ 
tract  again  requires  trade  studies  to  iden¬ 
tify  the  “trade  space,”  although  the  abil¬ 
ity  to  make  substantive  changes  decreases 
as  the  design  matures.  The  team  should 
employ  the  same  type  of  risk  program 
definition  as  described  in  Phase  I. 

Measurement.  The  team  continues  to 
track  TOC  estimates  to  TOC  objectives 
and  risk  drawdown  through  TPMs.  In 
addition,  the  contractor  EVMS,  in  con¬ 
junction  with  the  integrated  master  plan 
(IMP)  and  integrated  master  schedule 
(IMS),  provide  accurate  status  for  comple¬ 
tion  of  development  as  well  as  visibility 
to  any  potential  problems  before  they  get 
out  of  hand.  Data  from  prototype  and  com¬ 
ponent  builds  and  testing  will  support  pro¬ 
duction  cost  estimates.  The  team  will  have 


determined  the  availability  and  visibility 
of  such  metrics  during  their  partnering 
sessions. 

Phase  III 

In  Phase  III,  CAIV  can  generate  TOC 
savings  through  production  improvements 
prior  to  delivery  of  finished  systems  to  the 
user.  For  fielded  systems,  the  R-TOC 
process  is  primary  in  identifying  cost-sav¬ 
ing  opportunities,  but  the  CAIV  process 
really  starts  all  over  again  in  support  of 
system  modifications. 

Requirements.  The  IPT  ensures  con¬ 
tinued  ability  to  meet  baseline  require¬ 
ments  while  adapting  to  requirements 
evolutions  that  drive  system  modifications. 

Partnering.  Although  some  partners 
may  have  changed  from  earlier  phases 
(especially  if  major  modifications  are  in 
process),  the  same  principles  apply. 

TOC  focus.  With  an  eye  on  reducing 
immediate  costs,  contractors  are  given 
incentives  in  production  contracts  to 
streamline  manufacturing  processes  to 
reduce  the  cost  of  producing  systems. 
Additionally,  all  members  of  the  IPT, 
especially  maintainers,  work  to  ensure 
proposed  sustainment  processes  and 
practices,  maximize  system  availability, 
and  minimize  cost. 

Risk-based  management.  For  steady- 
state  systems,  teams  should  focus  on  risks 
that  may  upset  the  steady  state.  Risk 
occurs  in  all  phases  of  acquisition  and  it 
is  constantly  monitored  and  re-examined 
to  ensure  old  risks  are  managed  and  new 
risks  are  identified. 

Measurement.  As  in  previous  phases, 
a  well-functioning  EVMS  along  with 
manufacturing  IMP  and  IMS  provide  the 
IPT  the  necessary  information  to  assess 
program  progress  and  status. 
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Hybrids 

Most  acquisition  programs  consist  of 
modifications  and  ACAT  III  acquisition 
efforts.  One  of  the  most  innovative  aspects 
of  the  revised  acquisition  regulations  is 
the  recognition  that  these  systems  do  not 
need  to  go  through  the  classic  Phase  0, 1, 
II  sequence  before  being  fielded.  In  such 
cases,  the  IPT  should  review  the  above 
guidance  and  synthesize  or  tailor  to  the 
program  at  hand.  As  part  of  the  CAIV  pro¬ 
cess,  IPTs  are  encouraged  to  minimize 
acquisition  times  and  costs  by  stream¬ 
lining  the  acquisition  processes  wherever 
and  whenever  possible. 

Conclusion 


CAIV  is  a  viable  concept  for  attaining 
R-TOC  objectives.  CAIV  and  the  Air 
Force  R-TOC  process  go  hand-in-hand: 
CAIV  primarily  applies  to  systems  in  ac¬ 
quisition  and  the  R-TOC  process  applies 
to  fielded  systems.  Much  of  the  CAIV 
construct  described  here  is  not  new.  Parts 
of  it  have  been  applied  in  numerous  pro¬ 
grams.  The  authors,  and  many  other  pro¬ 
gram  managers,  really  used  key  elements 
of  CAIV  before  they  knew  it  was  CAIV. 
The  construct  presented  here  offers  a  more 
integrated,  definitive  CAIV  implementa¬ 
tion  description  than  has  previously  been 
available.  In  fact,  the  CAIV  construct  pre¬ 
sented  is  more  a  program  management 
construct  than  pure  CAIV. 

Challenges  to  realizing  the  benefits  that 
CAIV  offers  still  exist.  The  greatest  chal¬ 
lenge  is  the  need  to  make  decisions  based 
on  future  impacts  to  break  the  paradigm 
of  continuously  mortgaging  the  future 
when  faced  with  the  reality  of  the  critical 
exigencies  of  today.  That  paradigm  leads 


to  Under  Secretary  of  Defense  (Acquisi¬ 
tion  and  Technology)  Jacques  Gansler’s 
“death  spiral”  (1998).  Improved  under¬ 
standing  and  believable  quantification  of 
TOC  impacts  is  critical  to  overcoming  that 
fate.  The  key  targets  to  enable  better  in¬ 
formation  for  our  decision  makers  fall 
along  the  lines  of  the  CAIV  tenets: 

•  an  improved  requirements  process  to 
focus  on  capabilities-based  require¬ 
ments  supported  by  TOC  impact  data; 

•  improved,  partnered  management  of 
the  trade  space; 

•  recognition  by  every  member  of  the 
user,  acquirer,  and  contractor  team  that 
they  have  an  R-TOC  role  (in  addition, 
improved  cost  estimating  capabilities 
in  government  and  industry); 

•  improved  understanding  and  imple¬ 
mentation  of  risk  management; 

•  better  metrics; 

•  integrating  all  CAIV  aspects  for  effec¬ 
tive  program  management;  and 

•  training  users,  acquirers,  and  contrac¬ 
tors  at  all  levels  on  CAIV  implemen¬ 
tation  so  that  CAIV  becomes  ingrained 
in  the  culture. 

Finally,  CAIV  as  a  term  of  art  should 
disappear  in  the  future — but  everyone 
should  do  it!  In  1991,  a  senior-level  gov¬ 
ernment-industry  team  addressed  the 
problem  of  excessive  engineering  change 
proposals  in  development  by  defining 
Clear  Accountability  in  Design  (CAID). 
CAID  determined  that  the  government 
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would  not  take  configuration  control  be-  in  acquisition  today  cannot  identify  CAID, 
low  the  “A  Spec”  until  generally  after  but  it  is  the  standard.  CAIV  should  go  the 
physical  configuration  audit.  Most  people  same  way. 
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Ehdnote 


1.  The  contract  bid  was  a  $1.1  billion 
effort  for  control  software,  integra¬ 
tion,  test,  and  operational  activation 
of  the  $5  billion  command,  control, 
and  communications  system  for  inte¬ 
grated  air  defense  of  Saudi  Arabia. 
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Appendix 


Risk  Management  Approach 

Let  us  consider  an  example  of  a  poten¬ 
tial  risk  management  approach.  Typically, 
system  weight  is  a  critical  parameter  that 
will  impact  total  system  performance. 
Assume  that  keeping  weight  under  the 
allocated  threshold  is  particularly  vital,  as 
it  usually  is.  Numerous  studies  have  also 
shown  that  weight  typically  grows  through 
development  of  a  system.  Figure  6  shows 
an  allocated  threshold  along  with  “uncer¬ 
tainty  bands”  that  narrow  as  the  design 
matures.  Clearly,  a  tested,  validated  sys¬ 
tem  at  Milestone  (MS)  III  should  be  dis¬ 
tinctly  characterized.  However,  at  Mile¬ 
stone  I,  a  “paper  design”  only  estimates 
system  weight.  Assuming  the  weight 
Technical  Performance  Measure  (TPM) 


will  be  regularly  tracked,  we  can  define 
four  regions  on  the  chart  that  the  TPM  may 
be  in. 

I:  Weight  well  below  threshold;  high 
confidence  system  will  meet  thresh¬ 
old  at  MS  III. 

II:  Weight  below  threshold;  the  closer  to 
the  threshold,  the  more  active  efforts 
must  be  to  contain  growth. 

Ill:  Weight  above  threshold;  need  aggres¬ 
sive  weight  reduction  program. 

IV:  Weight  well  above  threshold;  low 
probability  to  meet  threshold  even 
with  aggressive  program. 
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Cost  Driver  Calculation 

Figure  7  is  a  simplistic  view  of  a  “comb 
chart”  analysis  of  cost  drivers  for  a  sys¬ 
tem.  The  costs  are  divided  into  RDT&E, 
procurement,  and  O&S.  Each  is  further 
subdivided  into  WBS  elements.  Such  sub¬ 
division  should  go  down  as  far  as  practi¬ 
cable.  For  each  element,  dollar  estimates 
and  percentages  of  TOC  are  generated. 
This  analysis  will  provide  a  baseline 


against  which  R-TOC  efforts  are  com¬ 
pared.  Currently,  SAF/AQCT  is  coordi¬ 
nating  10  Air  Force  pilot  programs  using 
this  methodology  to  generate  R-TOC 
plans  and  associated  objectives.  We  an¬ 
ticipate  that  this  procedure  will  be  directed 
on  almost  all  programs  in  the  future. 

Directions  for  conducting  such  analysis 
are  in  the  R-TOC  Guidebook  previously 
cited. 
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Figure  7.  Cost  Drivers 
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TUTORIAL 


PARTICIPATORY  CONTRACTING 


William  N.  Washington 

Participatory  contracting  represents  a  philosophy  in  which  the  government 
attempts  to  involve  outside  entities  in  a  form  of  partnership  or  coordinated 
effort,  with  the  goal  either  of  reducing  costs  or  improving  performance;  private 
industry  seeks  to  increase  profits  and  have  greater  control  over  the  effort.  This 
win-win  scenario  can  thus  appeal  to  all  participants,  and  makes  administration 
of  the  contract  more  of  a  partnership  effort,  for  its  success  benefits  all  the 
participants. 


Over  the  past  several  years,  the  fed¬ 
eral  contracting  world  has  seen 
several  changes,  as  the  government 
has  attempted  to  modernize  practices  and 
find  innovative  ways  to  improve  on  the 
procurement  process.  One  of  these  trends, 
which  I  shall  term  “participatory  contract¬ 
ing,”  is  to  involve  entities  outside  the 
government  in  a  form  of  partnership  or 
coordinated  effort.  Four  types  of  general 
contracting  seem  to  fall  into  this  type  of 
arrangement:  partnership  agreements, 
cooperative  research  and  development 
(R&D)  agreements  (CRADAs),  share-in¬ 
savings  (SiS),  and  research  tournaments. 


recent  Department  of  Defense  (DoD) 
guidance  to  promote  commercialization  of 
military  depots  (“Maintenance  of  Military 
Equipment,”  1996),  and  is  also  discussed 
in  a  Government  Accounting  Office  (GAO) 
Report  (1998).  Further,  these  types  of  ar¬ 
rangements  are  legal  under  Title  10  of  the 
United  States  Code.  Generally,  these 
efforts  have  fallen  under  four  types  of 
arrangements,  which  are  valued  at  about 
$500  million  annually  (Cahlink,  1999): 

•  direct  sales  (the  government  facility 
acts  as  a  subcontractor  for  private 
industry); 


Partnership  Agreements  at  Depots 


This  is  part  of  a  continuing  change  in 
function  for  the  depots,  in  that  the  depots 
are  currently  seeking  outside  work  in  or¬ 
der  to  better  utilize  existing  facilities,  and 
thus  reduce  costs.  This  is  in  keeping  with 


•  workshare  (the  program  manager  sends 
funds  directly  to  the  depot  for  part  of 
the  work,  and  contract  is  awarded  to 
private  industry  for  the  remaining 
portion); 

•  directly  contracting  out  the  repair  of 
military  equipment  to  private  industry, 
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where  the  depot  and  private  industry 
form  a  team  effort;  or 

•  leasing  the  space  or  facilities  at  depots 
to  private  industry. 

These  partnership  agreements  portend 
the  future  for  depot  facilities  and  usher  in 
the  concept  of  “commercialization”  of 
these  facilities,  which  in  turn  has  the 
potential  to  provide  both  small  and  major 
defense  firms  with  several  benefits,  such 
as: 

•  applied  engineering  programs; 

•  advanced  manufacturing  knowledge; 
and 

•  state-of-the-art  laboratory  or  manufac¬ 
turing  resources. 

These  efforts  further  bring  to  the  de¬ 
pots  outside  work  and  money  that  would 
lower  the  depots’  costs  by  fully  utilizing 
existing  personnel  and  facilities.  Further, 
through  working  with  other  government 
and  Service  programs  (such  as  the 
CRADAs  mentioned  below)  they  can  pro¬ 
mote  technology  transfer  and  areas  of 
science,  of  interest  to  the  military,  sup¬ 
porting  both  these  programs  and  the 
depots  (Washington,  1999). 

Cooperative  R&D  Agreements 

Cooperative  R&D  agreements 
(CRADAs)  have  been  used  for  teaming 
or  technology  transfer  projects  with  small 
businesses,  universities,  and  government 
laboratories. 


Teaming  and  technology  transfer 
projects  can  involve  commercial  work 
only,  projects  with  technology  transfer 
incubators  (TTI),  small  business  innova¬ 
tion  research  (SBIR),  and  small  business 
technology  transfer  (STTR).  These  ap¬ 
proaches  can  serve  as  a  bridge  between 
commercial,  government,  and  university 
R&D  and  production  applications  (Wash¬ 
ington,  1995).  They  can  also  incorporate 
existing  federal,  state,  and  local  funding 
initiatives  which  promote  small  businesses 
(in  1988  this  represented  $550  million  to 
promote  technology  innovation  [Peterson, 
1993])  to  help  provide  funding  for  the 
projects.  These  approaches  can  tie  into  the 
Services’  Centers  of  Excellence  Programs 
or  the  Office  of  the  Secretary  of  Defense- 
funded  university  research  initiatives. 

Further,  these  programs  have  been  ex¬ 
panding  somewhat  to  now  also  include 
joint  university-industry  research  projects 
(Gaumond,  1994).  The  Advanced  Re¬ 
search  Projects  Agency  (ARPA)  also 
funds  engineering  programs  through  its 
technology  reinvestment  project  (TRP) 
initiative,  in  conjunction  with  the  Nation¬ 
al  Science  Foundation  (Wax,  1995). 
Through  these  programs,  the  Services 
have  leveraged  the  best  universities  in  the 
nation  to  advance  the  state  of  science  in 
areas  of  interest  to  the  military  (and  also 
provide  external  funding  for  those  same 
projects).  These  approaches  are  thus  a 
win-win  scenario  for  new  technologies,  in 
that  they  promote  the  growth  and  devel¬ 
opment  of  new  firms  or  universities,  and 
provide  additional  research  and  develop¬ 
ment  on  technologies  of  interest  to  the 
military  at  reduced  costs  (being  partially 
subsidized  by  federal,  state,  and  local 
funding). 
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Share-in-Savings 


Another  type  of  initiative  contracting 
represents  a  variation  on  the  value  engi¬ 
neering  change  proposal  (VECP)  theme, 
which  is  similar  to  the  gain  sharing  ap¬ 
proach  used  in  private  industry.  The  Share- 
in-Savings  (SiS)  instrument  was  a  prod¬ 
uct  of  the  National  Defense  Authorization 
Act  of  Fiscal  Year  1996,  titled  “Share-in- 
Savings  Pilot  Programs”  (1998).  The  pro¬ 
gram  allows  for  the  use  of  saved  monies 
from  government  accounts  to  reward  a 
contractor(s)  for  successful  programs  and 
procedures  that  enable  the  government  to 
save  money.  The  use  of  SiS  in  government 
outsourcing  is  relatively  new,  and  since  it 
has  only  been  approved  for  limited  infor¬ 
mation  technology  pilot  projects  (10 
projects  between  $25  million  and  $100 
million,  and  another  10  projects  between 
$1  million  and  $5  million),  it  has  not 
received  much  attention. 

So  far,  three  projects  have  received 
Office  of  Management  and  Budget 
(OMB)  approval  for  an  SiS  pilot  program: 
the  Department  of  Energy’s  “Energy  Sav¬ 
ings  Performance  Contracts”  (1998);  the 
National  Aeronautics  and  Space  Admin¬ 
istration’s  “Shared  Savings  Clause” 
(1998)  used  for  the  modernization  of  their 
headquarters’  computer  networking;  and 
the  General  Services  Administration’s 
“information  technology  projects”  (Frank, 
1999).  The  premise  behind  SiS  is  to  allow 
a  contractor  to  share  in  either  internal  or 
collateral  savings  that  have  been  gener¬ 
ated  as  a  result  of  the  contractor’s  actions; 
similar  to  a  VECP.  However,  SiS  can  op¬ 
erate  on  a  higher  funding  level,  both  in 
terms  of  the  absolute  dollars  that  can  be 
awarded  ($100  million),  and  in  terms  of 


not  being  limited  to  internal  savings  from 
a  specific  department  or  program.  In  this 
type  of  program  the  contractor  makes  the 
capital  investment  needed  to  execute  the 
initiative,  then  shares  substantially  in  the 
savings  that  are  derived  (i.e.,  under  current 
initiatives  up  to  50  percent). 

An  additional  feature  of  this  type  of 
program,  unlike  VECPs,  is  that  actual  sav¬ 
ings  to  the  overall  agency  can  be  used  for 
the  award  payments  to  the  contractor,  un¬ 
like  the  normal  fiscal  rules,  where  those 
savings  would  have  to  be  returned  to  the 
general  trea¬ 


sury.  This  fea¬ 
ture  can  be  a 
definite  advan¬ 
tage  in  some 
circumstances, 
and  has  been 
much  sought  af¬ 
ter  by  program 


"The  [Share-in- 
Savings  Pilot 
Program]  allows 
tor  the  use  of 
saved  monies 
from  government 
accounts  to  reward 
a  contractor(s)  for 


managers  for  successful  programs 


and  procedures  that 
enable  the  govern¬ 
ment  to  save 


money 


## 


the  past  several 
years.  Similar-  j 
ly,  at  about  the 
same  time  as 
SiS  was  ap¬ 
proved,  Dr.  Kenneth  Oscar,  Deputy  As¬ 
sistant  Secretary  of  the  Army  (Procure¬ 
ment),  suggested  an  “Acquisition  Reform 
Incentives  Clause”  (1996).  This  clause 
represents  a  variation  on  the  VECP  theme 
for  use  on  nonhardware  item  contracts.  It 
would  work  like  a  VECP  proposal,  but 
reflects  the  larger  percentage  savings 
(based  on  a  five-year  reward  payout  sched¬ 
ule)  typified  in  the  SiS  initiatives,  with  a 
slightly  different  but  appealing  wrinkle, 
that  the  contractor’s  percentage  payment 
would  decrease  over  time. 


375 


Acquisition  Review  Quarterly— Fall  2000 


Research  Tournaments 


Research  tournaments  (Fullerton,  1995; 
Taylor,  1995;  Washington,  1997)  repre¬ 
sent  a  competition  process  that  is  struc¬ 
tured  like  an  auction,  with  the  winner 
awarded  a  “prize”  for  the  best  product. 
The  auction  component  consists  of  the 
participants  paying  a  fee  for  entering  the 


"This  process  pro¬ 
motes  innovation 
on  the  part  of  the 
offerors,  and  pro¬ 
vides  firmer  cost 
estimates  for 
equipment,  since 
costs  are  based 
upon  completed 
hardware  and  not 
conceptual  hard¬ 
ware  estimates." 


tournament, 
which  could  be 
used  to  defray 
the  cost  of  the 
prize,  or  offset 
the  cost  of 
conducting  the 
competition. 

The  govern¬ 
ment  commits 
to  pay  the  re¬ 
search  tourna¬ 
ment  winner  a 
set  amount  that 


is  verifiable  by 
the  courts,  and  must  be  awarded.  The  se¬ 
lection  of  the  winner  would  be  based  upon 
specified  priorities  (e.g.,  performance  or 
cost)  established  by  the  government, 
which  would  be  specified  in  the  request 
for  proposal,  so  that  the  competing  firms 
would  know  which  innovations  or  priori¬ 
ties  were  most  important  in  winning  the 
prize.  Finally,  each  firm  would  submit  its 
prototype  at  the  end  of  a  specified  period 
of  time,  for  the  government  to  evaluate 
and  subsequently  award  the  prize  for  the 
best  product.  Thus,  the  competition  would 
differ  from  a  patent  competition,  in  that  it 
would  select  the  most  innovative  design 
across  a  group  of  offerors  that  would  win, 
with  the  quality  of  the  design  stressed  over 
the  date  of  discovery. 


This  process  promotes  innovation  on 
the  part  of  the  offerors,  and  provides 
firmer  cost  estimates  for  equipment,  since 
costs  are  based  upon  completed  hardware 
and  not  conceptual  hardware  estimates. 
Rich  and  Janos  (1994)  also  point  out  that 
the  “beauty  of  a  prototype  is  that  it  can  be 
evaluated,  and  its  uses  clarified,  before 
costly  investments  for  large  numbers  are 
made.”  This  is  also  in  keeping  with  DoD 
Directive  5000.1  (1996),  which  stresses 
modeling  and  simulation  of  new  systems. 

An  additional  benefit  of  this  type  of 
procurement  is  that  it  should  require  less 
government  oversight,  since  the  offeror 
has  already  developed  the  item,  and  is 
offering  it  at  a  fixed  price  to  the  govern¬ 
ment.  Thus,  concerns  about  overseeing  de¬ 
velopment  and  production  costs  are  ne¬ 
gated.  Finally,  as  mentioned  above,  the 
contractors  could  specify  along  with  their 
proposals  what  they  consider  to  be  appro¬ 
priate  rewards  or  fees  for  additional  or 
alternative  performance  goals.  This  would 
allow  the  source  selection  authority  to  per¬ 
form  up-front  tradeoffs  and  assessments. 
To  date,  the  author  has  not  seen  any  re¬ 
search  tournaments  used  by  any  of  the 
Services,  but  the  National  Academy  of 
Engineering  workshops  have  recently 
endorsed  the  concept  (National  Academy 
of  Engineering,  1999). 

Summary _ 

These  various  types  of  contracting  af¬ 
ford  both  the  government  and  private  in¬ 
dustry  significant  benefits.  For  the  gov¬ 
ernment,  they  offer  the  potential  for  re¬ 
duced  costs  and  improved  performance; 
private  industry  receives  the  potential  for 
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increased  profits  and  more  control  over 
those  projects.  This  win-win  scenario  can 
thus  appeal  to  all  the  participants,  and 
makes  the  administration  of  the  contract 
more  of  a  partnership  effort,  for  its  suc¬ 
cess  benefits  all  the  participants.  Further, 
while  each  of  these  different  contracting 


vehicles  approaches  this  partnership  pro¬ 
cess  from  a  different  perspective,  they  all 
attempt  to  seek  optimum  performance 
through  assigning  the  work  in  the  “best 
division  of  labor”  between  the  government 
and  outside  agencies  to  achieve  their 
shared  goals. 


William  N.  Washington  is  an  operations  research  analyst  with  the  Office  of  the 
Deputy  Chief  of  Staff  for  Resource  Management,  HQ  CECOM,  Fort  Monmouth, 
NJ.  He  has  a  B.S.  degree  from  Kansas  State  University  and  an  M.S.  degree  from 
Trinity  University.  His  doctoral  studies  were  in  Measurement,  Evaluation  and 
Research  Design  at  Michigan  State  University.  He  is  a  graduate  of  the  Army 
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Assistant  Secretary  of  the  Army  (Financial  Management  &  Comptroller)  Author 
of  the  Year  Awards  (1 997, 1 998  &  1 999),  a  DA  Commander’s  Award  for  Civilian 
Service  (2000),  a  DA  Achievement  Medal  for  Civilian  Service  (1998),  and  the 
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ranging  from  education/training  to  contracting  issues  and  cost  savings.  A  mem¬ 
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ACQUISITION  REVIEW  QUARTERLY 
GUIDELINES  FOR  CONTRIBUTORS 


The  Acquisition  Review  Quarterly 
(ARQ)  is  a  scholarly  peer-reviewed  journal 
published  by  the  Defense  Acquisition  Uni¬ 
versity.  All  submissions  receive  a  masked 
review  to  ensure  impartial  evaluation. 

Submissions 


Submissions  are  welcomed  from  any¬ 
one  involved  in  the  defense  acquisition 
process.  Defense  acquisition  is  defined  as 
the  conceptualization,  initiation,  design, 
development,  test,  contracting,  production, 
deployment,  logistic  support,  modifica¬ 
tion,  and  disposal  of  weapons  and  other 
systems,  supplies,  or  services  to  satisfy 
Defense  Department  needs,  or  intended  for 
use  in  support  of  military  missions. 

Research  Articles 


Manuscripts  should  reflect  research  or 
empirically-supported  experience  in  one 
or  more  of  the  aforementioned  areas  of 
acquisition.  Research  or  tutorial  articles 
should  not  exceed  4,500  words.  Opinion 
pieces  should  be  limited  to  1,500  words. 

Research  articles  are  characterized  by  a 
systematic  inquiry  into  a  subject  to  dis¬ 
cover  or  revise  facts,  theories,  etc. 


Manuscript  Sections 


The  introduction  should  state  the  purpose 
of  the  article  and  concisely  summarize  the 
rationale  for  the  undertaking. 

The  methods  section  should  include  a 
detailed  methodology  that  clearly 
describes  work  performed.  Although  it  is 
appropriate  to  refer  to  previous  publica¬ 
tions  in  this  section,  the  author  should  pro¬ 
vide  enough  information  so  that  the  expe¬ 
rienced  reader  need  not  read  earlier  works 
to  gain  understanding  of  the  methodology. 

The  results  section  should  concisely 
summarize  findings  of  the  research  and 
follow  the  train  of  thought  established  in 
the  methods  section.  This  section  should 
not  refer  to  previous  publications,  but 
should  be  devoted  solely  to  the  current 
findings  of  the  author. 

The  discussion  section  should  empha¬ 
size  the  major  findings  of  the  study  and 
its  significance.  Information  presented  in 
the  aforementioned  sections  should  not  be 
repeated. 

Research  Considerations 


Contributors  should  also  consider  the 
following  questions  in  reviewing  their 
research-based  articles  prior  to  submission: 
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•  Is  the  research  question  significant? 

•  Are  research  instruments  reliable  and 
valid? 

•  Are  outcomes  measured  in  a  way 
clearly  related  to  the  variables  under 
study? 

•  Does  the  research  design  fully  and 
unambiguously  test  the  hypothesis? 

•  Did  you  build  needed  controls  into  the 
study? 

Contributors  of  research-based  submis¬ 
sions  are  also  reminded  they  should  share 
any  materials  and  methodology  necessary 
to  verify  their  conclusions. 

Criteria  for  Tutorials 


Tutorials  should  provide  special 
instruction  or  knowledge  relevant  to  an 
area  of  defense  acquisition  to  inform  the 
Defense  Acquisition  Workforce. 

Topics  for  submissions  should  rely  on 
or  be  derived  from  observation  or  experi¬ 
ment,  rather  than  theory.  The  submission 
should  provide  knowledge  in  a  particular 
area  for  a  particular  purpose. 

Opinion  Criteria 


Opinion  articles  should  reflect  judg¬ 
ments  based  on  the  special  knowledge  of 
the  expert.  Opinion  articles  should  be 
based  on  observable  phenomena  and 
presented  in  a  factual  manner;  that  is, 
submissions  should  imply  detachment. 


The  observation  and  judgment  should  not 
reflect  the  author’s  personal  feelings  or 
thoughts.  Nevertheless,  opinion  pieces 
should  clearly  express  a  fresh  point  of 
view,  rather  than  negatively  criticize  the 
view  of  another  previous  author. 

Manuscript  Style 


We  will  require  you  to  recast  your  last 
version  of  the  manuscript,  especially 
citations  (e.g.,  footnotes  or  endnotes)  into 
the  format  required  in  two  specific  style 
manuals.  The  ARQ  follows  the  author 
(date)  form  of  citation.  We  expect  you  to 
use  the  Publication  Manual  of  the 
American  Psychological  Association  (4  th 
Edition),  and  the  Chicago  Manual  of  Style 
(14th  Edition). 

Contributors  are  encouraged  to  seek  the 
advice  of  a  reference  librarian  in  complet¬ 
ing  citations  of  government  documents. 
Standard  formulas  of  citations  may  give 
only  incomplete  information  in  reference 
to  government  works.  Helpful  guidance  is 
also  available  in  Gamer,  D.L.  and  Smith, 
D.H.,  1993,  The  Complete  Guide  to  Citing 
Government  Documents:  A  Manual  for 
Writers  and  Librarians  (Rev.  Ed.),  Bethesda, 
MD:  Congressional  Information  Service, 
Inc. 


Copyright  Information 


The  ARQ  is  a  publication  of  the  United 
States  Government  and  as  such  is  not 
copyrighted.  Because  the  ARQ  is  posted 
as  a  complete  document  on  our  home  page, 
we  will  not  accept  copyrighted  articles  that 
require  special  posting  requirements  or 
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restrictions.  If  we  do  publish  your  copy¬ 
righted  article,  we  will  print  only  the  usual 
caveats.  The  work  of  federal  employees 
undertaken  as  part  of  their  official  duties 
is  not  subject  to  copyright. 

In  citing  the  work  of  others,  it  is  the 
contributor’s  responsibility  to  obtain 
permission  from  a  copyright  holder  if  the 
proposed  use  exceeds  the  fair  use  provi¬ 
sions  of  the  law  (see  U.S.  Government 
Printing  Office,  1994,  Circular  92:  Copy¬ 
right  Law  of  the  United  States  of  America, 
p.  15,  Washington,  DC:  Author).  Contribu¬ 
tors  will  be  required  to  submit  a  copy  of 
the  written  permission  to  the  Editor  before 
publication. 

Copyright  Policy 


We  reserve  the  right  to  decline  any  ar¬ 
ticle  that  falls  into  these  problem  copy¬ 
right  categories: 

•  The  author  cannot  obtain  official  per¬ 
mission  to  use  previously  copyrighted 
material  in  the  article. 

•  The  author  will  not  allow  DAU  to  post 
the  article  with  the  rest  of  the  ARQ 
issue  on  our  home  page. 

•  The  author  requires  that  unusual  copy¬ 
right  notices  be  posted  with  the  article. 

•  To  publish  the  article  requires  copy¬ 
right  payment  by  DAU  Press. 

Manuscript  Format 


Pages  should  be  double-spaced  and 
organized  in  the  following  order:  title 


page,  abstract,  body,  reference  list, 
author’s  note  (if  any),  and  figures  or  tables. 
Figures  or  tables  should  not  be  inserted 
(or  embedded,  etc.)  into  the  text,  but  seg¬ 
regated  one  to  a  page  following  the  text. 
If  material  is  submitted  on  a  computer  dis¬ 
kette,  each  figure  or  table  should  be  re¬ 
corded  in  a  separate,  exportable  file  (i.e., 
a  readable  .eps  file).  For  additional  infor¬ 
mation  on  the  preparation  of  figures  or 
tables,  see  CBE  Scientific  Illustration 
Committee,  1988,  Illustrating  Science: 
Standards  for  Publication,  Bethesda,  MD: 
Council  of  Biology  Editors,  Inc.  Please 
restructure  briefing  charts  and  slides  to  a 
look  similar  to  those  in  previous  issues  of 
ARQ. 

The  author  (or  corresponding  author  in 
the  case  of  multiple  authorship)  should 
attach  to  the  manuscript  a  signed  cover 
letter  that  provides  the  author’s  name,  ad¬ 
dress,  telephone  and  fax  number,  and  an 
e-mail  address.  The  letter  should  verify 
that  the  submission  is  an  original  product 
of  the  author;  that  is  has  not  been  pub¬ 
lished  before;  and  that  it  is  not  under  con¬ 
sideration  by  another  publication.  Details 
about  the  manuscript  should  also  be  in¬ 
cluded  in  this  letter:  for  example,  title, 
word  length,  a  description  of  the  computer 
application  programs,  and  file  names  used 
on  enclosed  diskettes,  etc.  A  short  biogra¬ 
phy  of  no  more  than  75  words  and  a  photo 
of  the  author  will  be  expected  from  each 
author.  Author  names  and  e-mail  addresses 
are  not  part  of  the  75-word  count. 

The  letter,  one  copy  of  the  printed 
manuscript,  and  any  diskettes  should  be 
sturdily  packaged  and  mailed  to:  Head¬ 
quarters,  Defense  Acquisition  University, 
Attn:  DAU  Press  (ARQ),  9820  Belvoir 
Road,  Fort  Belvoir,  VA  22060-5565. 
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In  most  cases,  the  author  will  be  notified 
that  the  submission  has  been  received 
within  48  hours  of  its  arrival.  Following 
an  initial  review,  submissions  will  be 
referred  to  referees  and  for  subsequent 
consideration  by  the  Executive  Editor, 
ARQ. 


Contributors  may  direct  their  questions 
to  the  Editor,  ARQ ,  at  the  address  shown 
above,  by  calling  (703)  805-3801  (fax 
805-2917),  or  via  the  Internet  at: 
blanch_norene@dsmc.dsm.mil 

The  DSMC  Home  Page  can  be  accessed 
at: 

http://www.dsmc.dsm.mil. 
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